How should security teams test LLMs against EU AI Act prohibited behaviors before deployment?

What should teams test before an LLM goes live?

Turn the legal prohibitions into explicit test cases, then check whether the model can be driven into banned outputs by ordinary prompts, indirect prompts, role-play, instruction injection, or layered jailbreaks. The test plan should be tied to the deployment context, because the same model may be acceptable in one workflow and unsafe in another. The goal is evidence of failure modes, not just a pass/fail label.

Testing should cover the prohibited behavior itself and the paths that can reach it. For an EU AI Act review, that means looking for social scoring logic, emotion inference, prohibited profiling, and illegal instruction generation, plus attempts to smuggle those outcomes through seemingly benign requests. Teams should also test whether the model refuses consistently when prompts are ambiguous, adversarial, or framed as policy analysis, fiction, or classification support.

Pre-deployment testing is stronger when it uses a structured governance lens. The EU AI Act regulatory framework sets the boundary conditions, while the test plan turns those obligations into observable behaviors you can exercise and record. That keeps the work focused on the actual prohibited practice rather than on generic model quality.

How do red-team tests expose prompt bypass and guardrail gaps?

Red-teaming should assume the model will resist direct prompts but may fail under pressure, context shifts, or multi-turn manipulation. Good test cases vary the wording, sequence, and framing to see whether the model still violates policy when a prohibited request is disguised as moderation, safety research, user profiling, or downstream automation. This is especially important when the model is wrapped in tools or agents, because outer controls can fail even if the base model appears safe.

Security teams should test both the model and the application layer around it. A model may refuse a direct prohibited request, yet still emit harmful content when asked to rewrite, summarise, translate, classify, or compare examples. They should also inspect whether safety filters can be bypassed by prompt chaining, role confusion, or output-format coercion, since those weaknesses often reveal where policy enforcement is too shallow.

For a practical testing baseline, use a generative-AI risk profile that treats pre-deployment red-teaming as a normal control, not an exceptional exercise. NIST AI 600-1 GenAI Profile is useful here because it frames testing, provenance, and incident readiness as part of release discipline, not as an afterthought.

What evidence should decide whether the model is fit for release?

Teams should keep a test log that shows the exact prompt, the model version, the system prompt or policy layer in effect, the output, and the decision made from that output. The most useful evidence is not that the model “usually refuses”, but that it reliably refuses the prohibited class across prompt variants and still behaves safely when adversarial phrasing is used. If results are inconsistent, the model is not ready for a broad release.

The release decision should also consider whether the model’s failure is isolated or systemic. A single bad answer may indicate a narrow prompt weakness; repeated failures across related prompts suggest a deeper alignment or control problem. Teams should pay close attention to outputs that look compliant on the surface but still enable prohibited use, because those are the cases most likely to survive casual review and reach production users.

For teams building a broader AI control stack, it helps to compare the testing result with a dedicated governance and control program. The NIST AI Risk Management Framework gives a useful structure for turning red-team findings into release gates, escalation criteria, and residual-risk acceptance decisions.

Risk and Threat Considerations

Prohibited-behavior testing matters because the failure is not just policy drift, it is exposure to outputs that can create legal, reputational, and operational harm before anyone notices. If the model can be steered into forbidden content through indirect or multi-turn prompts, the same weakness may also let users bypass other controls or elicit unsafe instruction paths.

Failure mechanism: The model or its surrounding guardrails apply the prohibition too narrowly, so prompt manipulation, context shifts, or tool-assisted workflows can produce outputs that still meet the banned condition even after an apparent refusal.

Impact: Unsafe output can reach users, downstream systems, or audit logs, creating compliance failure, loss of trust, and a higher chance that the same gap will be exploited again in production.

Practitioner Guidance

What to prioritise: Build a prohibited-behavior test suite from the legal text first, then expand it with adversarial variants that try to reach the same outcome indirectly. The hardest cases are usually not the obvious ones, but the prompts that cause the model to discuss, simulate, or normalize the banned behavior while appearing technically cautious.

What to verify: Confirm that refusal is consistent across direct requests, paraphrases, multi-turn pressure, and benign-looking proxy tasks. If the model only fails when the request is explicit, the guardrail is too brittle for deployment.

Practitioner takeaway: Treat pre-deployment AI safety testing as a control-validation exercise, not a content-quality review, and block release until the model can resist both the prohibited request and the common ways users try to smuggle it in.

Related resources from NHI Mgmt Group

• Guide: Agentic AI Compliance Guide
• How should security teams test AI guardrails before deployment?
• How should security teams test AI models for hidden backdoors before deployment?
• How should security teams test privileged access controls against AI agents before they are exposed to production systems?
• What happens if a business fails to prepare for the EU AI Act before deployment?