Should organisations prioritise model scoring or policy writing first for GenAI governance?

Policy writing matters, but scoring should come first when the objective is to understand how a model behaves under attack. Policies describe intent, while risk scores reveal whether the model can uphold that intent in practice. Without behavioural evidence, governance teams are approving assumptions rather than controls.

Why model scoring should come before policy writing

Scoring first gives governance teams evidence about how the model actually behaves, where it fails, and which attack paths matter in practice. Policy writing first can be useful for direction-setting, but without behavioural testing it risks codifying assumptions that have not been validated. For GenAI, the order matters because governance controls only work when they are anchored in observed model behaviour.

A score is not a substitute for policy, but it tells you whether the policy has a realistic target. If a model is fragile under prompt injection, over-answers unsafe requests, or leaks sensitive context, the policy has to address those behaviours directly rather than relying on abstract intent statements. That is why model evaluation belongs earlier in the governance workflow than the final drafting pass.

Model scoring also helps define scope. It separates issues that can be managed through usage rules from issues that require redesign, stricter approval, human review, or a different deployment pattern. Without that evidence, policy authors tend to write controls that sound complete but do not match the model’s actual risk profile.

How scoring and policy writing work together in GenAI governance

The best sequencing is iterative, not one-and-done. Start with scoring to establish baseline behaviour, then draft policy to reflect the actual control gaps you found, then rescore after tuning or guardrail changes. That cycle produces governance that is tied to measurable model performance rather than aspirational language.

Policy writing still matters because it sets ownership, acceptable use, escalation paths, and approval conditions. But those statements are stronger when they are shaped by evidence from the model itself. A policy that bans a behaviour you have not tested is weaker than one that explicitly addresses a failure mode you have measured.

This is especially important for changes that affect deployment decisions. If the score shows the model cannot reliably follow safety constraints under adversarial prompting, the governance answer may be to restrict use cases, require human oversight, or delay release until the model is improved. In other words, policy should formalise the decision that scoring enables.

What good GenAI governance looks like in practice

Effective governance treats model scoring as the control validation layer and policy as the operating rulebook. The scoring output should be understandable enough for risk owners to use, while the policy should be specific enough that engineering and product teams can implement it without guessing.

For practitioners, the key question is not whether both artefacts exist, but whether they are connected. A policy that is not informed by evaluation data becomes paperwork. A score that is not translated into policy becomes a report with no enforcement. The useful state is when each scoring finding maps to a concrete governance decision, such as restricted access, red-teaming frequency, monitoring thresholds, or approval criteria.

That approach also makes accountability clearer. When a model’s behaviour changes after a vendor update, prompt change, or tool integration, rescoring shows whether the prior policy still holds. If it does not, the policy should be revised; if it does, the score provides the evidence trail that supports continued use.

Risk and Threat Considerations

Governance breaks when policy language outruns model behaviour. If organisations write policy first and score later, they can approve models that appear controlled on paper but remain vulnerable to prompt injection, unsafe outputs, or policy evasion in real use.

Failure mechanism: The policy is based on intended behaviour, while the model is deployed based on untested assumptions. Attackers and failure conditions exploit the gap between declared rules and observed behaviour, especially where the model can be induced to ignore guardrails or expose sensitive context.

Impact: Teams may authorise higher-risk use cases, miss control weaknesses, and discover failures only after deployment. That increases the chance of misuse, operational incidents, and governance decisions that cannot be defended with evidence.

Practitioner Guidance

What to prioritise: Run behavioural scoring or evaluation before final policy approval, especially for use cases that can influence customer interactions, internal decisions, or access to sensitive data. If the model has not been tested against realistic misuse and adversarial prompts, treat the policy as provisional.

What to verify: Make sure every major policy claim has a corresponding test result, benchmark, or red-team observation behind it. If you cannot show that evidence, the policy is describing intent rather than controlling risk.

Practitioner takeaway: Start with scoring when you need to govern actual model behaviour, then use policy to convert those findings into enforceable operating rules.

Related resources from NHI Mgmt Group

• Should organisations prioritise external exposure or internal credential governance first?
• Should organisations prioritise workflow governance or model development first for healthcare AI?
• Should organisations prioritise AI inventory discovery or policy writing first?
• How does the consumer-secret-entitlement model help with governance at scale?