They should treat the event as an accountability failure and preserve the full decision context immediately. The priority is to retain logs, identify the model version and data inputs involved, and determine whether similar decisions were made under the same conditions. That evidence is essential for remediation and regulatory response.
When an AI decision cannot be explained, what should the organisation preserve?
The first obligation is to preserve the decision context before it disappears. That means keeping the full prompt or input set, model version, policy or guardrail state, retrieved data, downstream tool calls, timestamps, and the exact output that was acted on. If the decision may affect rights, safety, eligibility, or high-impact operations, treat it like evidence, not just troubleshooting data.
Explanation failure is often a traceability failure, so the preservation step should be immediate and tamper-aware. If logs roll over, telemetry is sparse, or the workflow only records the final answer, the organisation may lose the ability to reconstruct why the system behaved as it did or whether the same conditions produced other decisions.
Where the decision was produced by a governed AI workflow, the relevant question is not only “what did it say?” but “what state led to it?” That includes the model release, configuration, retrieval corpus, tool permissions, and any human override or approval path that was present at the time.
How do you determine whether the failure is isolated or systemic?
Once the evidence is preserved, the next step is to test whether the unexplained outcome was a one-off or a repeatable pattern. Review adjacent decisions made by the same model, under the same rules, on the same day, with the same input class, and with the same external data dependencies. A single unexplained result may indicate an edge case; repeated cases point to a control gap.
That review should look for consistency across similar records, not just identical inputs. If the system was using stale context, incomplete retrieval, a broken policy layer, or a changed model version, similar-looking decisions may have failed for the same hidden reason even if the outputs differ superficially.
It is also worth checking whether the explanation failure itself is a sign of missing observability. A decision process that cannot be reconstructed usually has gaps in logging, lineage, or change tracking that will affect more than one case.
What does accountability look like after an unexplained high-risk decision?
Accountability means assigning ownership for the decision path, the evidence trail, and the remediation. Someone must be able to say which service, team, or control owner can attest to the inputs, the version in use, and the retention of the record. Where the AI output influenced a material business action, the organisation should be able to justify that action to internal reviewers and external regulators.
For high-risk decisions, the standard should be higher than “the model produced an answer.” The organisation needs enough traceability to show how the answer was generated, who approved its use, and what checks existed to stop or flag unsupported outcomes. In practice, this often means treating explanation failure as a governance incident rather than a mere user experience problem.
This is also where the EU AI Act regulatory framework becomes operationally relevant, because high-risk AI obligations place real weight on traceability, record keeping, and human oversight. Organisations should align their evidence retention and incident handling so they can support those duties if a decision is later challenged.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST AI RMF and NIST SP 800-53 Rev 5 set the technical controls, while EU AI Act and ISO/IEC 42001:2023 define the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| EU AI Act | High-risk AI system obligations | High-risk AI decisions require traceability, record keeping, and governance over outputs. |
| Recommendation — Retain decision logs, model versioning, and oversight evidence for high-risk AI outputs. | ||
| NIST AI RMF | Govern map measure manage | Explained decisions depend on AI risk governance, measurement, and documentation. |
| Recommendation — Document inputs, outputs, and monitoring needed to explain consequential AI decisions. | ||
| ISO/IEC 42001:2023 | A.6.1 — AI risk management | AI management systems require controlled handling of AI risk and accountability evidence. |
| Recommendation — Maintain audit-ready records for high-risk AI decisions and their control context. | ||
| NIST SP 800-53 Rev 5 | AU-2 — Audit Events | Unexplained decisions require retained audit events to reconstruct what the system did. |
| AU-6 — Audit Record Review, Analysis, and Reporting | Reviewing similar decisions and spotting patterns depends on analysis of retained logs. | |
| Recommendation — Log the decision inputs, outputs, model version, and approval path as auditable events. Review related decision records to identify repeatable failure conditions. | ||
Practitioner Guidance
What to prioritise: Preserve the artefacts that let you reconstruct the decision before you investigate root cause. If the evidence is not retained immediately, the post-incident analysis becomes speculative and the accountability problem becomes harder to contain.
What to verify: Confirm the exact model version, input set, retrieval context, tool activity, and policy state for the affected decision. Then verify whether the same configuration produced other high-risk decisions in the same operating window.
Decision rule: If a high-impact decision cannot be explained from the retained record, treat it as a control failure until proven otherwise. The practical response is to escalate, freeze similar decisions if needed, and review the approval path before relying on the system again.
Practitioner takeaway: In high-risk AI, an unexplained decision is not just missing commentary, it is missing assurance. The organisation’s job is to preserve enough context to prove what happened, determine whether it happened elsewhere, and decide whether the control environment is still trustworthy.