Who is accountable when verified AI research is used to test real applications?

Why This Matters for Security Teams

Accountability is the control that decides whether verified AI research becomes a managed security activity or an unmanaged risk. When a team is authorised to test real applications, the key question is not who can technically run the test, but who owns the decision, the boundaries, and the consequences. That ownership should be explicit in policy, approvals, and evidence handling, because research activity can quickly affect production systems, customer data, and incident response workflows.

Security teams often get this wrong by treating verification as a blanket permission rather than a scoped exception. Good governance means the organisation defining the scope also owns logging, containment, escalation, and remediation follow-up. This is consistent with NIST SP 800-53 Rev 5 Security and Privacy Controls, which expects clear responsibility assignment across control families, not informal trust in the researcher. In practice, many security teams encounter unclear accountability only after a test result has already been used in a way that exceeded the original approval.

How It Works in Practice

In operational terms, accountability should follow the chain of authorisation. The organisation that approves the activity should define the test case, the environment, the permitted techniques, the data boundaries, and the stop conditions. If the work involves an agentic system, verified access by a researcher does not transfer operational ownership of the system under test. The research may be legitimate, but the application owner, security lead, and governance function still need to answer for how the work was conducted and what happened after.

A practical process usually includes:

• Written scope and purpose, including what is in bounds and out of bounds.
• Named approvers for the research plan, not just informal verbal consent.
• Logging of actions, outputs, and any evidence collected during testing.
• Rules for handling exploits, proofs of concept, secrets, or sensitive findings.
• Decision points for disclosure, remediation, and re-testing.

For identity and access control, this aligns well with NIST SP 800-63 Digital Identity Guidelines, because verified access should support strong authentication and assurance without weakening operational accountability. If the testing touches privileged paths or sensitive credentials, organisations should also treat the researcher as a tightly governed subject of access, not as an exception to normal control design. Verified AI research is most defensible when it is logged, time-bounded, and reviewable by the same governance process that would govern any other high-risk change.

These controls tend to break down when research is moved into live production without a single accountable owner because approvals, evidence, and rollback decisions become fragmented across teams.

Common Variations and Edge Cases

Tighter approval and review controls often increase coordination overhead, requiring organisations to balance faster testing against stronger assurance. That tradeoff is especially visible when the research is time-sensitive, cross-functional, or tied to a live incident.

There is no universal standard for every scenario, but current guidance suggests the following distinctions. If the work is internal red-teaming, accountability usually sits with the organisation’s security leadership and risk owner. If an external researcher is involved, the contracting entity still keeps accountability for scope, oversight, and outcome handling. If the activity is framed as AI safety evaluation rather than security testing, governance may expand to include model risk, legal review, and privacy oversight.

Special care is needed when the testing could expose customer records, trigger abuse of an AI agent, or interact with production automation. In those cases, the organisation should decide in advance who can halt the test, who can approve disclosure, and who must sign off on remediation completion. Where the work involves broader cyber risk management, CISA Secure by Design is a useful reminder that accountability should be built into the system and process, not retrofitted after a finding. The same principle applies to AI testing: verified access may legitimise the activity, but it does not dilute ownership of harm, misuse, or incomplete remediation.

Related resources from NHI Mgmt Group

• Who is accountable when shadow AI is used from managed endpoints?
• Who is accountable when a verified identity is later used for fraud?
• Who is accountable when an AI integration is used to create administrative access?
• Why do static tests miss the real risks in generative AI applications?