How should security teams evaluate image jailbreak risk across vision-language models?

Security teams should assume that a jailbreak tuned to one vision-language model will usually stay local to that model and not generalise broadly. The practical test is cross-model evaluation, especially across checkpoints, training data variants, and closely related architectures. The study suggests transfer becomes more plausible only among highly similar models, so risk assessments should focus on model families, not just the attack technique.

Why Model Families Matter More Than the Attack Label

Image jailbreak risk is best assessed as a property of a specific vision-language model family, not as a generic “image jailbreak” technique. The same prompt-image pairing can behave very differently across checkpoints, training mixes, and nearby architectures, which means a single successful bypass says little about broader exposure. Security teams should therefore treat cross-model transfer as the real question, then test whether the behaviour persists only within closely related models or spreads farther.

That distinction matters because many evaluation programmes overstate risk by assuming transfer where there is none. When a jailbreak is tuned to one model’s alignment stack, preprocessing, or visual grounding quirks, the weakness often stays local. A useful comparison is model versioning, not just model class, because small changes in fine-tuning or safety post-processing can significantly alter susceptibility.

In practice, many teams discover that an “AI attack” is actually a model-specific failure only after they have already generalised the result across their entire estate.

How It Works in Practice

A defensible evaluation plan starts by holding the attack constant and varying the model. Use the same adversarial image set, then test across base models, instruction-tuned variants, safety-tuned variants, and closely related checkpoints. If transfer remains strong only among highly similar models, the result points to shared representational behaviour rather than a broadly portable jailbreak method.

That means the evaluation should answer three separate questions: does the jailbreak work on the target model, does it carry over to sibling models, and does it survive changes in the training or alignment pipeline? The third question is often the most operationally important, because small changes in fine-tuning, safety filtering, or multimodal fusion can break a previously effective exploit.

Keep the attack artefact stable so the comparison is about model behaviour, not attack redesign.
Group models by family, checkpoint lineage, and training variant before comparing results.
Record whether failures are complete, partial, or contingent on specific instructions or image features.
Separate “same-model success” from “cross-model transfer” in reporting, because they imply different risk.

Where possible, include models that differ by a single control variable, such as alignment tuning or visual encoder choice, so you can identify what actually changes susceptibility. These controls tend to break down when teams test only one flagship model and then infer estate-wide exposure from that single result.

Common Variations and Edge Cases

Tighter evaluation often increases testing cost, requiring teams to balance depth against the number of model variants they can practically cover. The main edge case is a model family that shares enough architecture and training behaviour for transfer to look real, but only under narrow conditions such as a specific image format, captioning path, or instruction style.

Another common variation is when a jailbreak appears to transfer because the surrounding application layer is consistent, not because the models themselves are equally vulnerable. In those cases, the risk sits partly in preprocessing, prompting, or downstream policy enforcement, so model-level conclusions need to be separated from pipeline-level conclusions. Best practice is evolving here, and there is no universal standard for how much similarity is “close enough” to count as meaningful transfer.

Teams should also be careful not to overread a negative result. A jailbreak that fails across unrelated models may still be serious if the target deployment relies on a narrow model family or a single vendor lineage. The practical question is whether the organisation’s actual model portfolio shares enough structure for the exploit to matter operationally.

Risk and Threat Considerations

Image jailbreaks create two risks: a false sense of broad compromise when the issue is model-local, and missed exposure when a narrow family-level weakness is treated as isolated. Attackers benefit when defenders assume transferability without testing, or when they assume a failure in one model says nothing about nearby variants.

Failure mechanism: The exploit usually depends on model-specific alignment behaviour, visual tokenisation, or instruction-following thresholds. If sibling models share those mechanisms, the same jailbreak may transfer; if they do not, the attack may fail outside the original target. That makes similarity, not technique name, the relevant threat boundary.

Impact: Overgeneralising a single jailbreak can distort prioritisation, waste remediation effort, or conceal a real family-wide exposure. Under-testing can leave a model portfolio vulnerable where shared training or architecture creates a repeatable bypass path.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST AI RMF, NIST IR 8596 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.

Framework	Control / Reference	Relevance
OWASP Agentic AI Top 10	A1 — Prompt Injection and Instruction Hijacking	Image jailbreaks are a multimodal instruction-hijacking problem for model behaviour.
Recommendation — Test whether adversarial inputs can override model instructions across related multimodal variants.
NIST AI RMF	MAP — Measure and Manage Risks	The question is about evaluating and comparing AI risk across model variants.
Recommendation — Measure jailbreak behaviour across model families and document where transfer stops.
NIST IR 8596	GENAI-3 — Input Manipulation and Evasion	Image jailbreaks exploit adversarial inputs to alter model outputs.
Recommendation — Assess multimodal input manipulation paths and validate controls against evasion attempts.
MITRE ATLAS	AML.T0002 — Prompt Injection	Adversarial image inputs can steer model behaviour through injection-like manipulation.
Recommendation — Map the jailbreak to adversarial input techniques and compare resilience across model families.
NIST CSF 2.0	GV.RM — Risk Management Strategy	Teams need a repeatable way to compare AI model risk and prioritise exposure.
Recommendation — Define a model-risk test plan that compares transferability across the deployed model portfolio.

Practitioner Guidance

What to prioritise: Rank evaluations by model lineage first, then by vendor, checkpoint, and multimodal stack. The most useful finding is not “does the jailbreak work?” but “how far does it transfer before it stops working?”

What to verify: Confirm that the same artefact was used across all tests and that any apparent transfer is not caused by a shared application wrapper, prompt template, or image preprocessor. If those layers are constant, separate their effect from the model’s intrinsic susceptibility before making a risk decision.

Practitioner takeaway: Treat image jailbreak testing as a family-comparison exercise, not a one-off exploit demo, because the operational question is where the weakness repeats, where it stops, and which nearby models still share the same failure pattern.