How do organisations explain past machine learning predictions for audit and compliance?

Why This Matters for Security Teams

Past predictions are often scrutinised after an adverse outcome, not during model development. For audit and compliance, the question is not simply whether the model was accurate overall, but whether a specific output can be traced to a documented model version, approved data set, and known operating conditions. That traceability sits alongside broader governance expectations in NIST SP 800-53 Rev 5 Security and Privacy Controls and the evidence discipline described in NIST Cybersecurity Framework 2.0.

Security, risk, legal, and model operations teams often get this wrong by treating explainability as a one-time model feature rather than an evidentiary process. A compliant explanation normally needs to answer who approved the model, what data shaped it, which features were used, what thresholds applied, and whether the serving environment matched the one tested. If the organisation cannot reconstruct those details, it may still have a functioning model, but it does not have defensible decision records.

In practice, many security teams encounter explainability only after a regulator, auditor, or customer dispute has already forced reconstruction of a prediction path.

How It Works in Practice

Operational explainability depends on preserving the full chain of custody from training to inference. That means logging the training data sources, feature definitions, model code or artifact hash, approval records, deployment timestamp, and the exact inference inputs and output. Where decisions are high impact, current guidance suggests pairing technical logs with human-readable rationale records so reviewers can understand the factors that influenced a result without reverse-engineering the model from scratch.

In mature environments, model governance is built around reproducibility. Teams typically retain:

• Model lineage, including versioning and provenance of data, code, and parameters
• Feature and threshold documentation, so a prediction can be interpreted in context
• Inference logs showing input values, confidence scores, and decision timestamps
• Change control evidence for retraining, tuning, and rollback events
• Approval and exception records for production release and periodic review

That evidence set supports both technical reconstruction and compliance review. It is also consistent with the control intent of ISO/IEC 27001:2022 Information Security Management and the implementation guidance in ISO/IEC 27002:2022 Information Security Controls, especially where organisations need documented operating procedures, access control, and retained records.

For regulated decisioning, explanation quality also depends on the use case. A credit or fraud model may need a concise reason code, while a safety, AML, or hiring model may need deeper traceability and independent review. The key is that the explanation should match the audience: auditors need reproducibility, investigators need lineage, and affected individuals may need a clear statement of the factors that mattered most. These controls tend to break down when model outputs are generated by ad hoc notebooks or unmanaged API wrappers because the serving path no longer matches the approved training record.

Common Variations and Edge Cases

Tighter explainability often increases operational overhead, requiring organisations to balance decision transparency against model complexity and release speed. That tradeoff is especially visible when the model is a black-box deep learning system or when the organisation uses third-party foundation models that do not expose full internal reasoning. In those cases, best practice is evolving, and there is no universal standard for how much internal explanation is enough for every regulatory context.

One common variation is the difference between local and global explanation. A global explanation describes how the model generally behaves, while a local explanation describes why one prediction was produced at one time. Audit teams usually care more about the local case, but governance teams need both. Another edge case appears when prediction results depend on live context, such as risk scores that change with external signals. If the data source is unstable, the explanation must capture the state at inference time, not just the underlying model.

In financial crime, KYC, and AML contexts, explanation records may also need to support FATF Recommendations — AML and KYC Framework aligned evidence handling, where investigators expect both traceability and defensible escalation. The same is true when an organisation later retrains a model: a newer explanation cannot substitute for the one tied to the historical decision. The hardest cases are legacy models with poor lineage, because the organisation then has to decide whether to rebuild the evidence trail, constrain use, or retire the system entirely.

Related resources from NHI Mgmt Group

• When does a machine identity become a compliance problem?
• Why do healthcare organisations need PAM for both compliance and patient safety?
• How should security teams govern non-human identities for compliance?
• Why do non-human identities create more audit risk than human accounts?