Community-Defined Eval

Expanded Definition

A community-defined eval is a context-specific assessment that measures how an AI system performs on workflows a particular community cares about, not just on abstract academic tasks. In practice, the community may be a security team, product group, research lab, or operations function that agrees on success criteria, failure modes, and scoring rules for its own use case. That makes the eval more useful than a generic benchmark when the question is whether the system can safely support a real workflow with real constraints.

The key distinction is that the benchmark reflects local operational meaning. A community-defined eval may test whether an agent produces a correct incident summary, preserves evidence quality, follows a defined escalation path, or completes a support task without unsafe tool use. Definitions vary across vendors and research groups, and no single standard governs this yet, so the strength of the eval depends on whether the rubric is transparent, reproducible, and tied to the workflow it claims to represent. For governance and risk management, the closest reference point is the NIST Cybersecurity Framework 2.0, which emphasises outcome-focused risk treatment rather than generic technical scoring.

The most common misapplication is treating a community-defined eval as a universal quality signal, which occurs when one team’s workflow-specific score is presented as proof of general model reliability.

Examples and Use Cases

Implementing community-defined evals rigorously often introduces maintenance overhead, requiring organisations to weigh better workflow fidelity against the cost of designing and updating the rubric.

• A security operations team creates an eval for LLM-assisted triage that scores whether the output preserves source evidence, names confidence limits, and avoids unsupported attribution.
• A software engineering group measures code assistant behaviour on the team’s own debugging workflow, including whether the model asks for missing logs before proposing a fix.
• A compliance function builds an eval around report drafting to test citation quality, traceability to source material, and whether the system resists inventing policy references.
• An agentic AI team evaluates tool-using behaviour against a staged workflow so it can measure whether the agent completes tasks in the right order without overstepping its authority.
• A fraud operations group defines an internal benchmark for case summarisation, checking whether the model separates signal from speculation and keeps human review points intact.

These examples show why community-defined evals often sit closer to operational assurance than to pure model science. They can also be aligned with the outcome-based logic used in the NIST Cybersecurity Framework 2.0, especially when the goal is to verify that a system supports a defined security or business process reliably.

Why It Matters for Security Teams

For security teams, community-defined evals matter because AI risk is usually workflow-shaped, not model-shaped. A model can look strong on general benchmarks and still fail in ways that matter operationally, such as omitting evidence, mishandling sensitive data, bypassing approval steps, or taking unsafe actions through connected tools. That is especially important for NHI and agentic AI security, where autonomous systems may interact with secrets, tickets, logs, APIs, and privileged workflows.

Well-designed evals help teams detect whether an AI system respects role boundaries, keeps humans in the loop where required, and behaves consistently under the conditions that exist in production. They also support procurement and governance decisions by making vendor claims testable against local requirements instead of marketing language. In broader cybersecurity governance, this aligns with the practical intent of the NIST Cybersecurity Framework 2.0, which is to measure whether controls achieve intended outcomes. Organisations typically encounter the limits of a community-defined eval only after an AI system passes a lab test but fails in live operations, at which point the benchmark becomes operationally unavoidable to address.

Related resources from NHI Mgmt Group

• What is the difference between a network perimeter and an identity-defined perimeter?
• Should organisations allow community MCP servers in production development environments?
• What breaks when MCP runs behind gateways without defined auth propagation?
• How do organisations know if their AI eval rubric is actually useful?