Notifications
Clear all
Topic starter
24/07/2026 4:10 pm
TL;DR: Evaluation results for AI systems are often sparse, opaque, and hard to compare, which lets models and agents look safer than they are, according to Noma Security’s contribution to the EvalEval Coalition’s EveryEvalEver initiative. Standardised metadata for provenance, generation settings, and instance-level failures turns evaluation integrity into a security control, not a reporting preference.
NHIMG editorial — based on content published by Noma Security: LLMjacking: How Attackers Hijack AI Using Compromised NHIs
By the numbers:
- A third runs the same benchmark but with an optimized prompt template that inflates results by 10+ points.
Questions worth separating out
Q: How do security teams know whether AI review outputs are actually trustworthy?
A: Teams need to validate the integrity of the entire observation chain, from repository files to the model’s context window.
Q: Why do average benchmark scores create risk for AI governance?
A: Average scores compress too much information.
Q: How can organisations compare AI evaluations across vendors or models?
A: They need standard metadata for provenance and generation settings.
Practitioner guidance
- Standardise evaluation provenance fields Require evaluator relationship, methodology, and verification status in every AI evaluation before it is used for deployment or procurement decisions.
- Record generation configuration for every test run Capture temperature, sampling parameters, prompt templates, and inference engine details so teams can compare results across runs without hidden drift.
- Review instance-level failure cases before approval Inspect the specific prompts, tool calls, or permission-boundary scenarios where a model or agent fails, rather than relying on the aggregate score alone.
What's in the full article
Noma Security's full article covers the operational detail this post intentionally leaves for the source:
- The schema fields that record evaluator relationship, generation configuration, and instance-level failure detail
- The validation tooling and how it is intended to make evaluation records interoperable across different systems
- The coalition context, including how the initiative is being adopted across the AI evaluation community
- The practical examples of agentic evaluation traces and tool-call metadata that matter during implementation
👉 Read Noma Security's analysis of EveryEvalEver and AI evaluation trust gaps →
AI evaluation trust gaps: what security teams need to change?
Explore further
24/07/2026 4:12 pm
Evaluation trust gap: AI governance now depends on metadata quality, not just benchmark volume. If evaluation results cannot be compared, provenance cannot be verified, and generation settings cannot be inspected, then deployment teams are making access decisions on unstable evidence. That is a governance failure, not a measurement nuisance. The practitioner conclusion is straightforward: no evaluation should influence production trust unless it is reproducible and attributable.
A question worth separating out:
Q: How should security teams govern agentic AI as it moves into production?
A: Security teams should govern agentic AI as a class of non-human identity, not as a generic application feature. That means assigning ownership, scoping permissions tightly, logging every tool action, and revoking access on a defined lifecycle. Production rollout should require clear approval points for high-risk actions and continuous monitoring for drift.
👉 Read our full editorial: AI evaluation trust gaps are becoming an AI security problem
