Most organisations now run AI systems and AI agents that make or shape decisions, yet few internal audit plans test them with the same rigour as financial or IT general controls. An AI audit is not a model accuracy test: it asks whether the organisation knows which AI it uses, who is accountable for it, what data and identities it relies on, how it is monitored, and whether the controls around it work. This guide explains how to scope an AI audit, which frameworks give usable criteria, how to test governance, data, models, agents and third-party AI services, and how to report and follow up. It is written for chief audit executives, IT and technology audit leads, and the risk and security teams who will sit on the other side of the table.
Key takeaways
- Start with the inventory. An AI audit cannot give assurance over systems nobody has listed, so the first test is whether a complete AI register exists, including embedded AI features in SaaS and agents built by business teams.
- Use published criteria, not opinion. The IIA AI Auditing Framework, the GAO AI Accountability Framework, the NIST AI RMF and ISO/IEC 42001 give auditable expectations; the EU AI Act adds legal obligations for high-risk systems.
- Agents are identities with permissions. Every AI agent acts through API keys, tokens or service accounts, so access, least privilege and logging tests from identity audits apply directly.
- Test operation, not existence. A policy, a model card or a risk assessment proves design; sampling decisions, logs, overrides and change records proves the control operates.
- Plan for change. Models, prompts and providers change faster than annual audit cycles, so build continuous indicators and change triggers into the audit plan rather than relying on a single point-in-time review.
What an AI audit covers
The scope is wider than data science. A useful AI audit universe includes four kinds of system, each with different evidence:
| AI type | Examples | Main audit questions | Typical evidence |
|---|---|---|---|
| Predictive and decision models built in house | Credit scoring, fraud scoring, demand forecasting | Is the model validated, monitored for drift and bias, and changed under control? | Validation reports, performance monitoring, model change tickets |
| Generative AI applications | Customer chatbots, internal assistants built on a hosted large language model (LLM) | Are prompt injection, data leakage and output handling risks assessed and tested? | Threat model, red team results, guardrail configuration, incident log |
| AI agents | Agents that call tools, APIs or other agents and take actions without a human per step | What can the agent do, under whose identity, with what approval, and is every action logged? | Agent register, permission sets, credential inventory, action logs, approval records |
| Third-party and embedded AI | AI features in SaaS platforms, vendor scoring services, AI coding assistants | Was the feature risk assessed before it was turned on, and do contracts cover data use and incidents? | Vendor due diligence, data processing terms, feature enablement records |
The audit is also distinct from the other assurance activities around AI. Model validation (often a second-line function in banks) tests whether a model is fit for purpose. AI red teaming tests how a system fails under attack. See the AI Red Teaming Guide. ISO/IEC 42001 certification audits test a management system against a standard. Internal audit sits above these: it evaluates whether governance, risk management and controls, including those other assurance activities, are adequate and effective. For the wider method of running cyber audits, see the Internal Audit for Cybersecurity Guide.
Frameworks that give audit criteria
Internal audit needs criteria the auditee has accepted or must meet. These sources are the most widely used; most organisations pick one as the spine and map the others to it.
| Source | What it is | How auditors use it |
|---|---|---|
| IIA AI Auditing Framework (2024 update) | IIA framework covering AI governance, management and the internal audit function, with a practitioner's quick-start checklist | Engagement structure and the internal audit function's own readiness; links to the Three Lines Model and the Global Internal Audit Standards |
| GAO AI Accountability Framework (GAO-21-519SP, June 2021) | Four principles: Governance, Data, Performance and Monitoring, each with key practices, questions and audit procedures | Ready-made audit questions and procedures for auditors and third-party assessors |
| NIST AI Risk Management Framework 1.0 (January 2023) | Voluntary framework with four functions: Govern, Map, Measure and Manage, plus the AI RMF Playbook | Criteria for the risk management process; NIST AI 600-1 (July 2024) adds generative AI risks |
| ISO/IEC 42001:2023 | Certifiable requirements for an AI management system (AIMS) with reference controls in Annex A | Management system criteria; clause-based testing; reliance on certification where scope fits |
| ISO/IEC 42006:2025 | Requirements for bodies that audit and certify AI management systems, supplementing ISO/IEC 17021-1 | Judging how much reliance to place on a supplier's or the organisation's ISO/IEC 42001 certificate |
| EU AI Act (Regulation (EU) 2024/1689) | Law with risk tiers and obligations for providers and deployers | Compliance testing for in-scope systems, such as deployer duties under Article 26 |
| ICO Guidance on AI and data protection | UK GDPR guidance on accountability, DPIAs, fairness, accuracy, security and individual rights in AI | Privacy criteria, including Article 22 solely automated decisions; the ICO states the guidance is under review following the Data (Use and Access) Act |
Two points matter when choosing. First, the NIST AI RMF is voluntary and NIST states that version 1.0 is being revised, so cite the version you audited against. Second, an ISO/IEC 42001 certificate is evidence about a management system within a defined scope; check the scope statement before relying on it for a specific system. See the NIST AI RMF Guide and the EU AI Act vs ISO/IEC 42001.
Building AI into the risk-based audit plan
- Build the AI audit universe: start from the AI register; if none exists, that is the first finding. Add sources the register usually misses: SaaS features switched on by administrators, model API keys in cloud and finance records, agent platforms, and coding assistants. See the AI Register and AI Inventory Template and the Shadow AI Discovery Guide.
- Rate each system: use the organisation's risk taxonomy. Useful drivers are decision impact on people, autonomy (does it act or only advise), data sensitivity, regulatory classification (for example EU AI Act high-risk), external exposure and the breadth of permissions it holds.
- Choose engagement types: an AI governance review every two to three years, system-level audits for high-risk models and agents, and AI questions inserted into existing audits (procurement, privacy, identity, change management).
- Map coverage to criteria: show which engagements cover which GAO principles or NIST AI RMF functions over the cycle, so gaps are visible to the audit committee.
- Coordinate assurance: use an assurance map of model validation, red teaming, privacy reviews and certification audits, then decide where to rely and where to test. See the Three Lines Model for Cyber Risk Guide.
- Check skills: AI audits need data, model and identity knowledge; co-source where needed, while the chief audit executive keeps responsibility for the opinion.
| Engagement | Scope | Suggested trigger or frequency |
|---|---|---|
| AI governance and strategy | Policy, roles, AI risk appetite, inventory, oversight committee | Every 2 to 3 years, or after major AI strategy change |
| High-risk model audit | One model end to end: data, development, validation, deployment, monitoring | Annually for models with material impact on people or finances |
| AI agent access and action audit | Agent identities, credentials, tool permissions, approvals, logs | Before production go-live and annually for agents that change data or move money |
| Third-party AI review | Due diligence, contracts, data use, feature enablement | As part of third-party audit cycle; on major vendor AI feature launches |
| Regulatory readiness | EU AI Act or sector obligations for in-scope systems | Before obligations apply, then periodic |
These frequencies are a starting point, not a standard; set them from your own risk ratings.
Testing governance and accountability
- Policy and appetite: confirm an approved AI policy exists, defines acceptable and prohibited uses, and links to the risk appetite statement. Test that the policy is applied: sample new AI use cases and check they went through the defined approval route. See the AI Governance Guide.
- Named accountability: every system in the register should have a business owner, a technical owner and a risk owner. Sample entries and confirm the named people know they own them.
- Oversight forum: read committee minutes for evidence of challenge, not just noting. Check that rejected or paused use cases exist; a committee that has approved everything may not be operating as a control.
- Impact assessment: for systems that affect people, check that a data protection impact assessment and, where in scope, an EU AI Act fundamental rights impact assessment were completed before go-live, and that their actions were tracked to closure.
- Literacy and competence: check training records for staff who build, approve or oversee AI. See the EU AI Act AI Literacy Obligations (Article 4) Guide.
Testing data, models and monitoring
The GAO framework's Data, Performance and Monitoring principles map neatly onto system-level testing.
| Area | Control to test | Audit procedure |
|---|---|---|
| Data provenance | Training, fine-tuning and retrieval data sources are documented with lawful basis and quality checks | Trace a sample of datasets to source records and approvals; check retention and deletion |
| Representativeness and bias | Bias and fairness tests defined, run and reviewed before release | Re-perform or inspect test results for protected characteristics; check sign-off of known limitations. See the Explainable and Fair AI Guide |
| Validation | Independent validation before production and after material change | Confirm independence of validator; compare validated version with deployed version |
| Change control | Model, prompt, system prompt, guardrail and provider changes go through change management | Sample production changes from deployment logs and match them to approved tickets |
| Performance monitoring | Drift, error and complaint thresholds with escalation | Inspect dashboards and alerts; confirm a breach of threshold led to action |
| Human oversight | Defined review points with authority to override | Sample decisions; check override rates and that overrides are recorded. See the Human-in-the-Loop for AI Agents Guide |
| Security testing | Adversarial testing against prompt injection, data leakage and excessive agency | Review scope and results of red team or testing; confirm findings tracked to closure. See the OWASP Top 10 for LLM Applications Guide |
A frequent gap is the prompt layer: organisations version model weights carefully but change system prompts and retrieval sources directly in production. Treat prompts, tool definitions and retrieval indexes as configuration items subject to change control.
Where identity fits: auditing AI agents
AI agents act through non-human identities: API keys for model providers, OAuth tokens to SaaS, cloud roles and service accounts. Most of the highest-impact agent failures are access failures, so the agent audit borrows directly from identity audit procedures.
- Identity per agent: each agent should have its own identity, not a shared service account or a developer's personal token. Test by tracing a sample of agent actions in logs back to a unique identity. See the Agent Identity and Access Guide.
- Least privilege: compare granted permissions with the tools the agent actually calls. Excess write or delete rights are findings. The PocketOS database deletion shows the pattern: a coding agent found an unrelated, broadly scoped token in the codebase and used it to delete a production volume and its backups.
- Credentials: check that model API keys and agent tokens are held in a secrets manager, are short-lived where the platform allows, and are rotated. The LLMjacking incidents show why: stolen cloud credentials and AI API keys were used to run models at the victim's expense. See How to audit OpenAI and Anthropic organisation API keys.
- Approvals for high-impact actions: confirm that destructive, financial or external-communication actions require a human or policy check outside the agent, and sample the approval records.
- Action logs: confirm every tool call is logged with agent identity, user on whose behalf it acted, inputs, outcome and timestamp, and that logs are retained. Deployers of high-risk systems under the EU AI Act must keep system-generated logs under their control for at least six months (Article 26(6)) unless other law sets a different period. See the AI Agent Audit Trails for Regulators Guide.
- Recertification: check that agent access is reviewed periodically like human access. See Agent Identity Recertification.
Testing third-party and embedded AI
- Enablement control: many SaaS AI features are switched on by an administrator setting. Test whether enabling one triggers a risk review, and sample admin audit logs for AI feature changes.
- Contract terms: check data use for model training, data location, sub-processors, incident notification and audit rights.
- Assurance reliance: where a supplier provides an ISO/IEC 42001 certificate, check the certification scope and that the certification body works under ISO/IEC 42006 and ISO/IEC 17021-1 accreditation. Where it provides a SOC 2 report, check whether the AI service is in scope.
- Supply chain: for open models and libraries, check provenance records and an AI bill of materials. See the AI Supply Chain and AI-BOM Guide.
Rating and reporting AI findings
Rate AI findings on the organisation's normal scale, but make the impact concrete: the number of decisions affected, the people affected, the data exposed or the actions an agent could take. Audit committees respond better to "this agent can delete production data with no approval" than to "AI governance is immature".
- Separate design from operation: report whether the control is missing (design) or present but not working (operation), as in any audit.
- Link to criteria: cite the specific GAO practice, NIST AI RMF function, ISO/IEC 42001 clause or legal article each finding fails.
- Follow up on evidence: close findings only on tested evidence, not management assertion, and re-test after model or provider changes.
- Give an overall view: an annual summary of AI assurance coverage helps the audit committee see which systems have been tested and which have not.
Detection and measurement
Continuous indicators let internal audit see risk building between engagements. Agree data feeds with the first and second lines rather than building them in audit.
- Register coverage: percentage of AI systems discovered by technical scans (API key usage, SaaS feature settings, agent platforms) that are in the AI register. Target close to 100%.
- Assessment before go-live: percentage of production AI systems with a completed risk and impact assessment approved before deployment.
- Agent privilege gap: percentage of agents whose granted permissions exceed the tools they used in the last 90 days.
- Unapproved change rate: model, prompt or provider changes found in deployment logs without a matching approved change record.
- Monitoring response: median days from a drift or bias threshold breach to documented action.
- Finding closure: percentage of AI audit actions closed by the agreed date on tested evidence.
Framework mapping
| Audit area | NIST AI RMF | NIST CSF 2.0 | NIST SP 800-53 Rev. 5 | MITRE ATLAS |
|---|---|---|---|---|
| Governance and accountability | Govern | GV.OV-01 | RA-3 Risk Assessment | Not applicable |
| AI inventory | Map | ID.AM-02 | CM-8 System Component Inventory | Not applicable |
| Testing and validation | Measure | ID.IM-01 | CA-2 Control Assessments | AML.T0051 LLM Prompt Injection |
| Monitoring | Manage | DE.CM-03 | CA-7 Continuous Monitoring | Not applicable |
| Agent action logging | Manage | DE.CM-03 | AU-2 Event Logging, AU-12 Audit Record Generation | AML.T0053 AI Agent Tool Invocation |
| Third-party AI | Govern | GV.SC-07 | SA-9 External System Services, SR-3 Supply Chain Controls and Processes | Not applicable |
Common failure patterns
- Auditing the policy, not the systems: the engagement confirms an AI policy exists and stops, without sampling a single deployed system.
- Register taken on trust: the audit accepts the AI register as complete, missing SaaS features, agents and API keys that technical discovery would have found.
- Agents treated as software, not identities: nobody tests what the agent's credentials can do, so a broadly scoped token goes unnoticed until it is used.
- Prompt and retrieval changes outside change control: the model version is controlled, but system prompts and retrieval sources change in production without review.
- Over-reliance on certificates: an ISO/IEC 42001 certificate or vendor attestation is accepted without checking that the system under audit is in its scope.
- Point-in-time opinion on a moving target: findings are closed, then a provider model update changes behaviour and nobody re-tests.
Practitioner checklist
- Confirm a complete AI register exists and test it against technical discovery sources.
- Choose one criteria spine (GAO, NIST AI RMF or ISO/IEC 42001) and map the others to it.
- Add AI governance, high-risk model and AI agent audits to the risk-based plan with stated triggers.
- Sample new AI use cases to test that approval, impact assessment and owner assignment happened before go-live.
- Trace a sample of agent actions to unique identities and compare granted permissions with tools used.
- Check that model keys and agent tokens are in a secrets manager, scoped and rotated.
- Match production model, prompt and provider changes to approved change records.
- Verify scope and accreditation before relying on any ISO/IEC 42001 certificate or vendor report.
- Agree continuous indicators with the first and second lines and review them between engagements.
- Close AI findings only on tested evidence and re-test after material model or provider changes.
Standards and references
- The IIA: The IIA's Artificial Intelligence Auditing Framework
- GAO: Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities (GAO-21-519SP)
- NIST: AI Risk Management Framework
- ISO: ISO/IEC 42006:2025 Requirements for bodies providing audit and certification of AI management systems
- EUR-Lex: Regulation (EU) 2024/1689 (Artificial Intelligence Act)
- ICO: Guidance on AI and data protection
- OWASP: Top 10 for LLM Applications 2025
Related NHI Mgmt Group resources: Internal Audit for Cybersecurity Guide · AI Governance Guide · Audit Readiness for AI Agents and NHIs · ISACA AAISM and AAIA AI Certifications Guide