Notifications
Clear all
Topic starter
17/08/2026 4:36 pm
TL;DR: Even perfect reasoning, prediction, and learning do not solve the value step in AI alignment, according to Pentera, because Arrow’s impossibility theorem and Gibbard-Satterthwaite show preference aggregation cannot be made simultaneously fair, universal, non-dictatorial, and strategy-proof. The real constraint is governance, not computation: humans must choose which axiom to relax.
NHIMG editorial — based on content published by Pentera: AI alignment has a governance ceiling, not just a technical one
Questions worth separating out
Q: Why can’t AI alignment solve conflicting human values by optimisation alone?
A: Because the underlying problem is not just finding a better ranking rule.
Q: How should organisations handle strategic manipulation in human feedback systems?
A: Design the feedback process as if participants will learn how to game it, because the mathematics says they can.
Q: What do security teams get wrong about AI alignment?
A: Security teams often treat alignment as a one-time model training issue, then assume deployment controls will hold the line.
Practitioner guidance
- Define the alignment trade-off explicitly Document which property your AI decision process is allowed to relax: universality, neutrality, non-dictatorship, or strategy-proofness.
- Treat feedback channels as incentive surfaces Assume users, raters, or operators may adapt their inputs once they understand the system.
- Separate objective setting from model operation Keep the definition of fairness, safety, and acceptable override logic under human policy control rather than burying it inside model training.
What's in the full article
Pentera's full article covers the mathematical detail this post intentionally leaves at the governance level:
- Step-by-step walkthrough of Arrow’s theorem and the decisive coalition argument
- The Gibbard-Satterthwaite manipulation theorem and why onto rules matter
- How the article connects value aggregation limits to RLHF and constitutional AI
- The closing interpretation of why humans remain the decision layer outside the model
👉 Read Pentera’s analysis of AI alignment limits and value aggregation →
AI alignment and value aggregation: where the technical ceiling appears?
Explore further
17/08/2026 4:42 pm
AI alignment exposes a governance ceiling, not a model-quality problem. Pentera’s argument matters because it shifts the question from whether a system can reason better to whether it can legitimately combine conflicting human values. Arrow’s theorem makes clear that no aggregation rule can preserve all desirable properties at once. For practitioners, the implication is that alignment is always a choice about which constraint to relax.
A question worth separating out:
Q: Who should own AI value-setting decisions in an enterprise?
A: Ownership should sit with the business or security governance function that is accountable for the outcome, not with the model alone. If an AI system influences policy, access, or safety decisions, the organisation needs named human accountability for the objective function and its exceptions.
👉 Read our full editorial: AI alignment has a governance ceiling, not just a technical one
