Model vulnerability discovery is the process of finding weaknesses in an AI model that can be exploited or that cause unsafe behaviour. It covers prompt injection susceptibility, data leakage paths, jailbreaks, and other failure modes that affect model integrity, access boundaries, or trust in model outputs.
What Model Vulnerability Discovery Looks For
Model vulnerability discovery looks for weaknesses in an AI model’s behaviour and boundaries, including prompt injection susceptibility, data leakage paths, jailbreak resistance, and other failure modes that can distort or subvert model output.
For practitioners, the key point is that the target is not only “bad answers,” but any exploitable condition that changes what the model reveals, follows, or refuses under adversarial or malformed input.
Why These Weaknesses Matter
Weaknesses in a model can undermine trust in the system’s outputs, break intended access boundaries, and expose sensitive data through responses, latent instructions, or indirect leakage. When discovery is shallow, organisations often validate only the happy path and miss behaviours that appear only under adversarial prompting or chained interactions.
That is why model vulnerability discovery is closely tied to red teaming, abuse-case testing, and exposure review, not just model quality testing. It asks where the model can be made to ignore policy, reveal protected context, or behave in ways that make downstream controls less effective.
Common Vulnerability Patterns
The most common findings usually fall into a few broad classes. Prompt injection can cause the model to follow untrusted instructions over system intent. Jailbreaks can weaken safety constraints. Data leakage can occur through memorisation, retrieval paths, logging, or overly permissive tool use. Boundary failures can appear when a model is allowed to act on information or commands it should not see.
These patterns often overlap. A jailbreak may also create a leakage path, and a leakage path may reveal enough context to make later prompts more effective. For that reason, discovery should treat the model as part of a larger interaction surface, not as an isolated text generator.
How Organisations Use Discovery Results
Discovery findings are most useful when they feed design changes, guardrail tuning, and retesting. Teams should translate each weakness into a concrete control question: does this failure arise from the prompt layer, the model, retrieval content, surrounding application logic, or the way the system handles context and memory?
That same output also supports prioritisation. Findings that expose sensitive data, undermine access boundaries, or can be chained into broader compromise deserve faster treatment than issues that only affect polish or content quality. A structured AI Security Posture Management (AI-SPM) Guide can help connect discovered weaknesses to asset visibility and control ownership.
Risk and Threat Considerations
Model vulnerability discovery matters because adversaries can turn small behavioural gaps into practical abuse, especially when a model is connected to tools, data sources, or privileged workflows. The main risk is not only incorrect output, but exploitation of trust, disclosure of protected information, or manipulation of model-assisted decisions.
Failure mechanism: An attacker uses crafted prompts, injected content, or adversarial sequences to steer the model away from intended policy, reveal hidden context, or operate beyond its safe boundary.
Impact: The result can include data leakage, policy bypass, unreliable outputs, and unsafe downstream actions that become harder to detect once the model’s behaviour looks plausible.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack and risk surface, while NIST AI RMF and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | GOVERN — Govern | Model vulnerability discovery is an AI risk governance activity. |
| Recommendation — Establish governance for model testing, findings triage, and remediation ownership. | ||
| OWASP Agentic AI Top 10 | ASI06 — Memory & Context Poisoning | Prompt injection and context manipulation are core model vulnerability patterns. |
| ASI03 — Identity & Privilege Abuse | Model weaknesses often matter because they expand what an agent or model may access or do. | |
| Recommendation — Test for context-poisoning paths and harden prompt and memory boundaries. Constrain delegated privileges and review any tool-access expansion after findings. | ||
| NIST SP 800-53 Rev 5 | SI-10 — Information Input Validation | Discovery covers adversarial inputs that change model or application behaviour. |
| AU-2 — Event Logging | Discovery and retesting depend on preserving evidence of unsafe model behaviour. | |
| Recommendation — Validate and constrain untrusted model inputs before they influence decisions. Log prompts, outputs, and relevant context needed to reproduce failures. | ||
Practitioner Guidance
What to watch for: Treat vulnerability discovery as an ongoing testing discipline, not a one-time model review. The strongest signal is any weakness that changes what the model can see, say, or trigger under untrusted input.
Practitioner note: Record findings in a way that preserves the exact prompt, context, and model state that produced the weakness, then rerun the same case after each mitigation so you can tell whether the control actually changed the behaviour.