Grading Rule

How grading rules work

A grading rule defines what “correct” means for a system output. In benchmarked systems, that can mean an exact match, a set match, a substring or containment test, or agreement with an executable result, and the chosen rule directly changes how the same output is scored.

That matters because grading is not just a technical detail, it is part of the evaluation design. A strict exact-match rule rewards one answer shape, while a looser containment rule may accept multiple valid phrasings, so the rule can influence model behaviour, leaderboard ranking, and how easily different systems can be compared.

Common grading rule patterns

Grading rules usually fall into a few practical patterns. Exact comparison checks whether the output is identical to a reference answer. Execution-based grading runs code or a query and compares the result. Containment-based grading checks whether required values appear somewhere in the output, even if the surrounding format differs.

Each pattern is useful for a different task shape. Exact comparison is simple and reproducible, but it can penalise equivalent answers that differ only in formatting. Execution-based grading is better for tasks where the real requirement is a correct computed result. Containment rules are common when the important question is whether key facts are present rather than whether the whole response is fixed text.

Why the chosen rule changes outcomes

The same submission can pass under one rule and fail under another. For example, a response that contains the right values but adds extra text may succeed under a containment rule and fail under an exact-match rule. That means the rule is part of the benchmark contract, not an afterthought.

Because of that, grading rules shape incentives. A narrow rule can encourage brittle, format-sensitive answers. A broader rule can make benchmarks more tolerant of legitimate variation, but it may also hide errors if the inclusion criteria are too permissive.

Designing and interpreting grading rules

Good grading rules align with the actual goal of the evaluation. If the task is about precise machine-readable output, the rule should enforce structure tightly. If the task is about semantic correctness or retrieval of required facts, the rule should measure the right evidence without overfitting to one presentation style.

Practitioners should read benchmark results with the rule in mind, because a score is only meaningful relative to the acceptance criteria that produced it. Two systems can appear close on a leaderboard while being judged by very different rules, which makes the scores comparable only when the grading logic is understood.

Risk and Threat Considerations

Grading rules can create evaluation risk when they are misaligned with the real task, too permissive, or too brittle. That can produce false confidence, reward output shaping over actual correctness, and make benchmark results look stronger or weaker than the system truly is.

Failure mechanism: If the rule accepts the wrong answer shape, or rejects legitimate variation, then the evaluation can be gamed by formatting tricks, overfitting, or narrow optimisation to the test rather than the underlying objective.

Impact: The result can be misleading performance claims, poor model selection decisions, and a benchmark that fails to reflect real-world usefulness or reliability.

Practitioner Guidance

Why practitioners should care: Treat the grading rule as part of the system design, not just the scoring script. The rule should match the output contract the benchmark is actually trying to validate, otherwise the leaderboard will optimise the wrong behaviour.

What to watch for: Check whether the rule is overly exact, overly permissive, or sensitive to formatting differences that do not matter to the task. If the score changes materially when the same answer is reformatted, the grading logic may need review.

Related resources from NHI Mgmt Group

• What is the difference between behavioural analytics and traditional rule-based monitoring?
• Why does the 72-hour breach reporting rule matter for IAM and security teams?
• How should security teams govern bulk sensitive data transfers under the DOJ rule?
• How should crypto platforms implement Travel Rule compliance without creating excessive operational overhead?