How should AI safety teams evaluate domain-specific refusal behaviour?

Use domain-separated test sets that distinguish offensive misuse, authorised defensive work, and unrelated harmful content. A model that refuses one class but not another is not uniformly safe or unsafe, so evaluation has to measure boundary precision, spillover, and whether task framing changes the output pattern.

Why refusal evaluation has to be domain-separated

Domain-specific refusal behaviour is only interpretable when the test set separates threat categories that look similar at the surface but serve different purposes. A model that declines offensive misuse while still supporting defensive analysis is behaving differently from one that rejects both. That is why evaluation should measure the boundary, not just the average refusal rate.

The core issue is framing sensitivity. If a model flips its answer based on whether the same task is described as attack planning, blue-team analysis, or harmless discussion, the refusal policy is probably tracking wording too loosely. Good evaluation therefore checks whether the model responds to the underlying intent, the content itself, and the surrounding safety context, rather than to a single surface label.

That separation also improves diagnosis. When the model fails, teams need to know whether the miss came from over-refusal, under-refusal, or inconsistent treatment of adjacent cases. Without that split, a high refusal rate can hide poor discrimination, and a low refusal rate can hide selective unsafe compliance.

How to structure the test set and score the boundary

A practical set usually needs at least three buckets: offensive misuse, authorised defensive work, and unrelated harmful content. The first bucket tests whether the model blocks explicitly malicious intent. The second tests whether it still supports legitimate security work, such as analysis, detection, or hardening. The third checks whether it refuses harmful content that is not actually a security use case.

Microsoft Azure OpenAI abuse by Storm-2139 is a useful reminder that refusal evaluation cannot stop at obvious malicious prompts, because real abuse often uses legitimate-looking access paths and then pivots into harmful output generation.

Teams should score more than one number. Boundary precision asks whether the model refuses the right class and allows the right class. Spillover asks whether refusal on one domain contaminates nearby legitimate requests. Framing sensitivity asks whether the same intent gets different treatment when it is phrased as a security assessment, a policy exercise, or a harmful instruction. Those three signals together are more informative than a single pass or fail metric.

It is also worth testing paired prompts that differ only in intent or role. A strong evaluation will include near-neighbours, such as offensive exploitation versus defensive validation, because that is where false boundaries usually appear. If the model treats all cyber content as disallowed, it is not actually distinguishing risk, it is just suppressing the topic.

What good practice looks like for refusal analysis

Refusal evaluation should be designed like a measurement system, not a content filter. The aim is to see whether the model can make stable distinctions under realistic variation, especially when the same task is phrased in a benign, defensive, or malicious way. That means the test harness should preserve the core task while varying only the framing, so you can tell whether the refusal policy is actually grounded.

OWASP API Security Top 10 is a good analogue for this kind of thinking, because many failures come from boundary mistakes rather than from obviously broken behaviour. The lesson carries over: evaluate the narrow control point where a model is supposed to distinguish authorised from unauthorised use.

One useful operational check is to compare refusal behaviour across increasingly specific prompts. If a model refuses a broad harmful request, then later complies once the same request is embedded inside a supposedly defensive frame, the policy boundary is probably too brittle. Conversely, if it refuses every version, it may be safe in the narrow sense but unusable for legitimate safety work.

That is why human review still matters for the edge cases. Automation can tell you where the boundary moved, but practitioners still need to judge whether the boundary is aligned with the intended policy. The most important output is not “the model refused,” but whether it refused for the right reason, at the right granularity, and only where refusal was actually warranted.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

OWASP API Security Top 10 addresses the attack and risk surface, while OWASP ASVS and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.

Framework	Control / Reference	Relevance
OWASP ASVS	V2 — Validation and Business Logic	Tests whether prompt framing changes model behavior in a way that indicates boundary failures.
Recommendation — Validate framed and near-neighbour prompts separately to catch policy boundary mistakes.
OWASP API Security Top 10	API5 — Broken Function Level Authorization	Maps to distinguishing allowed defensive use from disallowed harmful use at the function boundary.
Recommendation — Separate authorised defensive functions from forbidden misuse in your evaluation set.
NIST CSF 2.0	DE.CM-01 — Networks and network services are monitored to detect potential cybersecurity events	Supports monitoring model outputs for refusal drift, spillover, and abnormal compliance patterns.
Recommendation — Monitor refusal patterns for drift and investigate unexpected compliance clusters.

Practitioner Guidance

What to verify: Check that each test item has a clearly stated intent label, because ambiguous prompts make refusal metrics meaningless. The best corpus mixes exact-harm prompts, clearly defensive prompts, and lookalike prompts that differ only by framing.

What to measure: Track false refusals on authorised defensive work separately from unsafe compliance on offensive prompts. If you only report aggregate refusal rate, you will miss the failure mode that matters most to deployment.

Common mistake: Treating all security-related content as one bucket. That shortcut produces inflated safety scores and hides whether the model can actually distinguish misuse from legitimate analysis.

Practitioner takeaway: The evaluation target is not universal refusal, it is accurate discrimination. A useful model refuses when it should, supports defensive work when it can, and stays consistent as the framing changes.