Gemini 4 in offensive security labs: what practitioners should expect

TL;DR: Gemini 4’s evaluation across six controlled web application labs found it was strongest at vulnerability discovery and reconnaissance, but its exploit reproduction dropped by 9 percentage points because of higher soft-refusal rates, according to Tenzai research. The result is that offensive capability is not yet the same thing as dependable offensive execution.

Editorial analysis by NHI Mgmt Group, based on content published by Tenzai: “Gemini 4: A New Hope”.

Key questions

Q: How should teams evaluate autonomous security models when discovery is stronger than exploitation?

A: Evaluate them in two stages.

Q: What does a high soft-refusal rate mean for agentic security workflows?

A: It means the model may stop or divert from a task even when it has enough context to continue.

Q: Why does tool-use behaviour matter in autonomous security assessment?

A: Tool-use behaviour shows how the model handles uncertainty, verification, and task sequencing.

Practitioner guidance

• Define separate success criteria for discovery and validation Score autonomous security tests in two phases: finding plausible weaknesses and reproducing them reliably.
• Track soft-refusal rate as a control metric Record how often the model stops, detours, or declines to continue during assessments, then compare that rate across tasks and model versions.
• Review tool-call mix before operational use Look at whether the system relies mainly on file reads, browser actions, shell execution, or code execution, because that pattern affects traceability, momentum, and the kind of evidence the agent can produce.

Bottom line: Gemini 4’s strongest result is in vulnerability discovery, but its exploit reproduction is less reliable because of higher soft-refusal behaviour.

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →