What do organisations get wrong about AI auditing in regulated environments?

A common mistake is treating AI auditing as a narrow technical check on the model alone. The FCA discussion instead points to broader auditing across model, data, governance, and documentation, with human review where needed. Organisations also miss the need for standards that are fluid enough to keep up with rapid AI development and varied use cases.

What organisations misunderstand about AI auditing

Organisations often approach AI auditability as if the model were the only asset that matters. In regulated settings, that misses the broader control surface: training and inference data, governance decisions, human oversight, documentation, change control, and evidence retention. If any of those are opaque, the audit fails even when the model itself appears well tested.

A second error is treating audit readiness as a one-time exercise. AI systems change quickly, use cases evolve, and assurance expectations shift with them, so a static checklist becomes stale fast. The practical standard is not “can we inspect the model once?” but “can we explain, evidence, and reproduce key decisions over time?”

Audit scope should include data lineage, model governance, approvals, exceptions, and post-deployment monitoring.
Human review matters where the system affects regulated decisions or where the control expectation is not fully automatable.
Evidence has to be kept in a form that supports reconstruction, not just internal confidence.

For a useful reference point on how regulated audit expectations expand beyond a narrow technical review, see the SOC 2 Trust Services Criteria (AICPA) and the EU AI Act.

Why audit failures happen in practice

The most common failure mode is overconfidence in model testing and underinvestment in governance evidence. Teams may be able to show performance metrics, but not who approved the system, what data was used, which version was deployed, or how exceptions were handled. In regulated environments, that missing chain of custody is often what regulators and internal auditors care about most.

Another recurring issue is that audit controls are built around the current version of the system rather than the operating process. That creates a gap when the model is retrained, the prompt or policy layer changes, a vendor component is swapped, or the business expands the use case into a higher-risk decision path. The audit story has to survive change, not just launch.

Practitioners also underestimate how much documentation quality affects audit outcomes. Sparse decision logs, unclear ownership, and vague human-review criteria usually matter more than whether the model uses one algorithm or another. If the control cannot be demonstrated by an independent reviewer, it is not a reliable control.

What good AI auditing looks like in a regulated setting

A defensible audit program separates model performance testing from broader assurance over governance and operations. That means the organisation can show the data sources, approval path, validation approach, monitoring triggers, and the conditions under which a human must intervene. It also means the audit pack can be refreshed as the system evolves, instead of being rebuilt from scratch.

The strongest programmes define evidence ownership early. Product, risk, compliance, and engineering should each own a different slice of the record, so no single team becomes the bottleneck for audit response. When that division is clear, audit questions are answered faster and with less reliance on ad hoc reconstruction.

For regulated AI, the point is not to produce more paperwork. It is to make the control environment legible enough that an external reviewer can understand why the system is acceptable, where the residual risk sits, and what would force a pause or escalation.

Keep versioned records for models, prompts, policies, data sources, and approval decisions.
Define escalation thresholds for model drift, policy exceptions, and high-impact use cases.
Ensure human oversight is specific, documented, and actually used where required.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

NIST CSF 2.0 and NIST AI RMF set the technical controls, while ISO/IEC 42001:2023 and EU AI Act define the regulatory obligations.

Framework	Control / Reference	Relevance
NIST CSF 2.0	GV — Govern	AI auditing in regulated settings is a governance problem spanning accountability, oversight, and evidence.
	PR.DS — Data Security	The answer depends on data lineage, provenance, and controls over training and inference data.
	DE.CM — Continuous Monitoring	Regulated AI auditing must keep pace with changes, drift, and post-deployment control failures.
Recommendation — Define governance ownership and accountability for AI audit evidence across the system lifecycle. Protect and track the data inputs that shape audited AI decisions and outcomes. Monitor AI systems continuously so audit evidence reflects current behaviour and risk.
NIST AI RMF	GOVERN — Govern	AI auditing requires structured governance over accountability, documentation, and oversight.
	MAP — Map	The subject requires identifying the AI system, its context, and risk impacts before audit design.
	MEASURE — Measure	Auditability depends on measuring performance, drift, and control effectiveness over time.
Recommendation — Establish AI governance processes that make auditability and accountability explicit. Map the AI use case, stakeholders, data, and decision impacts before setting audit scope. Measure model behaviour and control performance so audit evidence stays current.
ISO/IEC 42001:2023	A.6 — Planning AI risk treatment and objectives	AI auditing in regulated environments depends on planned controls, objectives, and recorded decisions.
	A.7 — Support	The answer stresses documentation, competence, and retained evidence for audit readiness.
Recommendation — Set AI governance objectives and risk treatments that produce auditable evidence. Maintain documented procedures and evidence so AI controls can be independently reviewed.
EU AI Act	Article 9 — Risk management system	Regulated AI auditing must show ongoing risk management, not just one-off testing.
	Article 12 — Record-keeping	The answer highlights the need for traceable logs and reproducible evidence in audits.
Recommendation — Operate a documented risk management system for AI throughout its lifecycle. Keep logs and records that let auditors reconstruct AI decisions and changes.

Practitioner Guidance

What to prioritise: Build the audit trail around decision-making and change, not just model testing. If you can explain one deployment but cannot reproduce the evidence after the next retrain or policy change, the control is too brittle for a regulated environment.

What to verify: Confirm that an auditor can trace each material outcome back to the governing approval, the data used, the active version, and any human intervention. If any of those links depend on tribal knowledge, the audit process is weaker than it appears.

Practitioner takeaway: The real test is whether the organisation can defend the system as an evolving governed process, not merely as a technically validated model.