Individual Conditional Expectation shows how one model input changes a single prediction while other inputs stay fixed. In machine learning risk analysis, it reveals case by case model behavior, exposing hidden variation, bias, and sensitivity that average methods can hide. Security teams use it to inspect decision consistency across identities, events, or controls.
How ICE works
Individual Conditional Expectation, or ICE, breaks a model’s prediction down to the level of a single case. By varying one feature at a time while holding the rest fixed, it shows how sensitive that specific prediction is to each input.
This matters because averages can hide important local behaviour. Two records may have the same model score overall, while ICE reveals that one is highly sensitive to a control, event, or attribute that barely affects the other.
What ICE reveals about model behaviour
ICE is useful when security teams need to understand whether a model behaves consistently across different identities, events, or control states. It helps surface non-linear responses, thresholds, interactions, and outlier cases that a global summary can smooth away.
It is especially valuable when a model is used in risk analysis or decision support, because a model can look stable in aggregate while still reacting very differently for individual cases. That difference is often where hidden bias, brittle logic, or unintended sensitivity lives.
Why ICE matters in security analysis
In security work, the point is not just to know that a model performs well on average, but to know where it may fail in specific conditions. ICE can help reviewers examine whether a signal behaves differently for certain users, systems, or events, which is important when model output influences investigation priority or access-related decisions.
It also helps distinguish genuine model behaviour from artefacts introduced by correlated inputs. If one feature appears important only for a subset of cases, ICE can show whether that effect is real, unstable, or dependent on another hidden factor.
ICE and its limits
ICE is an interpretation tool, not a proof of correctness. It shows how a model responds under constrained what-if changes, but those changes may create unrealistic combinations of inputs if the features are strongly related in the real world.
For that reason, ICE works best alongside domain review and other explanation methods. It can expose where the model is sensitive, but it cannot by itself tell you whether the sensitivity is appropriate, fair, robust, or operationally safe.
Risk and Threat Considerations
ICE is often used because model decisions can conceal concentrated error, bias, or instability that only appears for certain records. That creates risk when a security or governance team assumes the average explanation represents every case, especially where decisions affect access, prioritisation, or trust.
Failure mechanism: A model may appear consistent in aggregate while behaving unpredictably for specific combinations of inputs, so harmful sensitivity, proxy bias, or brittle thresholds go unnoticed until those edge cases are encountered.
Impact: Reviewers may miss unfair or unsafe model behaviour, mis-rank cases, or rely on a model whose local decisions are not robust enough for operational use.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST AI RMF and NIST CSF 2.0 set the technical controls, while ISO/IEC 42001:2023 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | Map, Measure, and Manage AI Risks | ICE directly supports AI model interpretation and risk measurement |
| Recommendation — Use model explainability results to measure and manage AI decision risk. | ||
| NIST CSF 2.0 | ID.RA-01 — Asset Vulnerabilities Are Identified and Recorded | ICE helps identify where model behaviour varies or breaks down for specific cases |
| GV.RM-01 — Risk Management Strategy Established | ICE informs how organisations judge model uncertainty and local failure modes | |
| Recommendation — Document model sensitivity findings as part of risk assessment. Use ICE outputs to inform model risk treatment and governance decisions. | ||
| ISO/IEC 42001:2023 | 4.1 — Understanding the organization and its context | ICE supports contextual understanding of how AI behaves in specific conditions |
| Recommendation — Incorporate explainability evidence into AI governance and assurance. | ||
Practitioner Guidance
What to watch for: Use ICE when you need to understand whether a model’s behaviour is stable across different records, not just whether the overall score looks acceptable. It is most useful when a single prediction can have security or operational consequences and you need to know what drives that specific outcome.
Practical note: Treat ICE as a diagnostic lens, then validate the patterns it exposes against real-world plausibility and the data relationships you expect. If the curve suggests a surprising jump or reversal, investigate whether the model is learning a legitimate rule or a spurious shortcut.