Dangerous Capability Evaluation is a test of whether a model can perform harmful actions, such as generating malware, phishing content, or exploit guidance. It helps organisations decide what safeguards and access limits are needed. Defenders evaluate the model before deployment, use red-team prompts, and gate high-risk capabilities behind stricter controls.
What this evaluation actually tests
Dangerous capability evaluation asks whether a model can carry out harmful tasks well enough to create real security exposure, not just produce unsafe-looking text. The test is about practical misuse potential, such as whether the system can generate convincing phishing lures, malware concepts, or exploit instructions that a defender would treat as operationally dangerous.
Because the subject is capability measurement, the main question is not whether a model can ever produce a risky sentence in isolation, but whether it can sustain the quality, specificity, and procedural coherence needed for abuse. That makes the evaluation a governance tool for deciding when stricter safeguards, access limits, or deployment constraints are justified.
How organisations use the result
Teams typically use dangerous capability evaluation before deployment, during red-teaming, and when materially changing model access or guardrails. The result helps separate low-risk systems from those that need tighter approval flows, stronger monitoring, or restricted use in higher-trust environments.
The practical value is that it turns an abstract concern into an operational decision. If a model can reliably assist harmful workflows, the organisation can treat that as a trigger for stronger controls rather than waiting for an incident to prove the point.
What makes the capability dangerous
The danger is not limited to direct cyber abuse. A model that can draft malware, sharpen phishing content, or explain exploitation steps can reduce attacker effort, improve scale, and lower the skill needed to execute harmful activity.
That matters because the model may accelerate the full abuse chain, from initial reconnaissance and social engineering through to payload development or post-compromise guidance. Even when outputs are partial, the practical effect can be to speed up iteration and make harmful work more accessible.
Why the evaluation needs controlled conditions
Dangerous capability testing is only meaningful when prompts, scoring, and success criteria are designed around realistic misuse scenarios. A weak test can miss harmful capability; an overbroad one can flag ordinary technical competence as dangerous, which leads to poor policy decisions.
The strongest programs combine red-team prompts, scenario-based scoring, and deployment gates so that the result maps to a real control decision. That lets organisations distinguish between a model that merely answers technical questions and one that can reliably support harmful action at useful quality.
Risk and Threat Considerations
Models with dangerous capabilities can reduce the cost of abuse for both opportunistic actors and more skilled adversaries. If they are exposed too broadly, they may be used to scale phishing, improve exploit development, or help attackers iterate faster than defenders can detect and respond.
Failure mechanism: The model produces actionable harmful guidance, and the surrounding product or workflow fails to constrain who can reach it, what it can generate, or how often it can be queried.
Impact: Organisations can see faster fraud attempts, more convincing social engineering, increased malware or exploit development support, and higher downstream risk to users, credentials, and systems.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack surface, NIST AI RMF and NIST CSF 2.0 set the technical controls, and ISO/IEC 42001:2023 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI02 — Tool Misuse | Covers harmful model-assisted actions through tool-like execution paths. |
| Recommendation — Restrict agent tool paths that could execute harmful instructions or abuse external systems. | ||
| NIST AI RMF | GV.1 — Govern AI | Supports governance decisions for evaluating and controlling risky AI capabilities. |
| Recommendation — Establish governance criteria that gate deployment of models with harmful capability. | ||
| MITRE ATLAS | Adversarial Machine Learning | Provides threat patterns for evaluating harmful AI abuse and red-team scenarios. |
| Recommendation — Map harmful-output scenarios to adversarial AI techniques and test for abuse paths. | ||
| ISO/IEC 42001:2023 | AI management system | Defines organisational AI governance, risk treatment, and control oversight. |
| Recommendation — Embed dangerous capability testing into the AI management system and approval process. | ||
| NIST CSF 2.0 | GV.OV-01 — Oversight of cybersecurity risk management | Aligns with oversight of risky model capability and control decisions. |
| Recommendation — Review model capability findings in governance oversight before release. | ||
Practitioner Guidance
Why practitioners should care: Treat the evaluation as a deployment control, not a research exercise. Its purpose is to inform whether the model can be placed into production with normal access, or whether it needs tighter gating because its outputs are too useful for abuse.
What to watch for: Pay close attention when a model consistently produces stepwise harmful guidance, adapts to red-team prompts, or maintains quality across multiple attempts. Those patterns usually matter more than isolated unsafe completions.
Practitioner takeaway: Use the result to decide where human review, stricter access, or feature limits are needed before the model reaches users who could turn capability into harm.