Expected utility is the average value of an action or choice when each possible outcome is weighted by its probability and utility. In AI and decision models, it is used to compare options under uncertainty, including agent actions that trade off reward, risk, cost, and constraint violations.
How expected utility works
Expected utility turns uncertain outcomes into a single comparable value by multiplying each outcome by its probability and utility, then summing the results. That makes it a decision rule, not a prediction rule: it helps choose between options when outcomes differ in both value and likelihood.
In practice, utility is the score assigned to an outcome, and it can reflect reward, loss aversion, cost, time, safety, or any other preference that matters to the decision-maker. Two options with the same average payoff can have very different expected utilities if one creates much larger downside risk or violates constraints more often.
Expected utility in AI decision-making
In AI systems, expected utility is often the bridge between a model that estimates outcomes and an agent that must choose an action. The model may assign probabilities to several possible future states, while the utility function encodes what the system is trying to optimise, such as success rate, resource efficiency, or reduced operational risk.
This is why expected utility shows up in planning, reinforcement learning, and automated decision policies. It gives a formal way to compare actions under uncertainty, especially when the “best” action is not the one with the highest raw reward but the one that balances reward against failure cost, latency, or policy violations.
Why probability and utility both matter
Probability alone cannot rank choices, because a highly likely outcome may still be undesirable. Utility alone cannot rank choices under uncertainty, because a desirable outcome may be unlikely or carry unacceptable variance. Expected utility combines the two so the decision reflects both likelihood and consequence.
That combination is especially important when a system must trade off average gain against tail risk. A choice with a slightly lower average return may still be better if it avoids rare but severe negative outcomes, or if it better satisfies a hard operational constraint.
Limits and practical interpretation
Expected utility is powerful, but it depends on how utilities are defined. If the utility scale does not match the real preferences of the business or the agent owner, the result can look mathematically optimal while still being operationally wrong.
It also assumes the probabilities and utilities are good enough to support comparison. In real systems, uncertainty can be poorly estimated, utilities can change over time, and hidden constraints may not be represented explicitly, so expected utility should be treated as a decision aid rather than a guarantee of the best real-world outcome.
Risk and Threat Considerations
Expected utility can become a security concern when the utility function rewards the wrong thing, underweights rare failures, or ignores adversarial behaviour. In AI and automated decision systems, attackers can exploit these blind spots by steering the system toward actions that look optimal under the model but are unsafe in practice.
Failure mechanism: If probabilities are manipulated, missing, or systematically biased, the decision rule can prefer an action that is locally optimal in the model but weak against abuse, distribution shift, or adversarial inputs.
Impact: The system may amplify risky actions, miss constraint violations, or create predictable failure patterns that reduce trust, resilience, and control.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST AI RMF and NIST CSF 2.0 set the technical controls, while ISO/IEC 42001:2023 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | Map, Measure, and Manage AI Risks | Expected utility is a core AI decision concept under uncertainty and risk trade-offs. |
| Recommendation — Use AI risk processes to validate that utility functions reflect safety, cost, and constraint trade-offs. | ||
| NIST CSF 2.0 | GV.RM-01 — Risk Management Strategy | Expected utility requires a decision strategy for weighing reward against risk and loss. |
| Recommendation — Align decision scoring to a documented risk strategy before automating choices under uncertainty. | ||
| ISO/IEC 42001:2023 | A.5.2 — AI policy | Utility-driven AI decisions should be governed by organisational AI policy and objectives. |
| Recommendation — Define policy limits for automated decisions so utility optimisation stays within approved risk boundaries. | ||
Practitioner Guidance
Why practitioners should care: Expected utility is only as good as the utility function behind it. If the scoring model does not encode safety, compliance, and operational constraints correctly, an agent can make decisions that are mathematically consistent but business-incompatible.
What to watch for: Review whether the objective function overvalues short-term reward, ignores tail losses, or treats hard limits as soft preferences. Where uncertainty is high or the cost of failure is asymmetric, decision policies need explicit guardrails rather than a plain average-value optimiser.
Related resources from NHI Mgmt Group
- When does just-in-time access reduce risk less than expected?
- How should security teams control vendor access in water utility OT environments?
- What breaks when authorization decision logs leave the expected jurisdiction?
- How should utility companies centralize authorization without breaking operations?