GPT-5.5 raises the bar for offensive AI security workflows

At a glance

What this is: XBOW’s evaluation shows GPT-5.5 improving offensive security workflows by finding more known vulnerabilities, completing login tasks faster, and behaving more effectively inside agent-based pentesting runs.

Why it matters: For IAM and security teams, the practical issue is not model novelty but governance: AI systems that can log in, pivot, and persist inside real workflows change how access, testing, and containment decisions need to be designed.

By the numbers:

• GPT-5 missed 40% of vulnerabilities.
• Opus 4.6 reduced that to 18%.
• GPT-5.5 brings it down further to just 10%.
• GPT-5.5 achieves 97.5%

👉 Read Xbow's analysis of GPT-5.5 performance in offensive security workflows

Context

Offensive AI testing is no longer limited to isolated prompt quality or single-task accuracy. The real governance question is how far model capability extends once it is embedded inside multi-step workflows that can discover vulnerabilities, authenticate into target systems, and decide whether to persist or pivot. That shift matters to security teams because it turns an LLM from a component into an operational actor inside the testing process.

For identity and access teams, the bridge is straightforward. Any system that can repeatedly attempt logins, interact with applications, and sustain decision-making across tasks starts to resemble a governed non-human workflow, even when it is being used defensively. That raises questions about credential handling, task scoping, and escalation boundaries. The starting point here is atypical only in scale, not in direction: most enterprises are moving toward more AI-assisted security operations, not fewer.

Key questions

Q: What breaks when AI security tools rely on model benchmarks alone?

A: Benchmark-only decisions break when teams assume capability scores translate directly into operational security. In practice, a strong model can still generate false positives or miss issues if the harness is weak, and a smaller model can outperform it inside a better workflow. The control problem is system design, not just model selection.

Q: Why do AI agents complicate access governance more than ordinary automation?

A: AI agents complicate access governance because they can branch at runtime, wait on external services, and continue later with the same operational context. That means privilege is not just granted at launch, it persists across a live session that must be observable, resumable, and attributable.

Q: How do teams know if AI-assisted pentesting is actually working?

A: Look for higher-quality findings, faster triage, and fewer unresolved false positives, not just more output. If the workflow still requires manual cleanup to make findings usable, the tool is adding noise rather than improving decision quality. Effective testing should shorten the path from discovery to verified action.

Q: What should security teams do when AI tools can log into live systems?

A: They should place those tools under the same access governance used for other privileged non-human workflows. That means task-scoped credentials, explicit target allowlists, logging, and termination rules when the tool drifts from the intended path. Live access should never be broader than the test objective requires.

Technical breakdown

Why agent workflows change model evaluation

Model benchmarks that test single responses do not capture how an LLM behaves inside a workflow that includes discovery, authentication, exploitation, and reporting. Once the model is placed in an agent loop, success depends on sequencing, state management, and error recovery, not just answer quality. That is why miss rate becomes a more useful measure than isolated correctness. It reflects whether the system can progress through a real task without dropping out early or getting stuck on a partial path.

Practical implication: evaluate AI tools on full task chains, not isolated prompts, before trusting them in offensive or defensive security work.

Black box vs white box testing in AI-assisted pentesting

Black box testing simulates an attacker with no source code, while white box testing adds code visibility and usually increases detection depth. XBOW’s result suggests the capability gap is shrinking because the model performs strongly even without source code, then scales further once code is available. That compression matters operationally because source access no longer guarantees a simple linear improvement. It also means the model can extract more value from the same target context than earlier generations could.

Practical implication: treat code disclosure as an amplifier, not a complete control, and assume AI-assisted testing will exploit both runtime and source-level clues.

Persist or pivot is now a security control decision

In agentic workflows, persistence is not always a virtue. A model that keeps pushing a failing path can waste time, create noise, or miss better avenues, while a model that gives up too easily can leave coverage gaps. The article shows improvement in this balance, which is a sign that workflow governance matters as much as model capability. The real control question is how much autonomy the agent should have when it encounters blocked access or incorrect credentials.

Practical implication: define explicit retry, abort, and escalation rules for AI agents interacting with sensitive systems.

Threat narrative

Attacker objective: The objective is to increase the speed and depth of vulnerability discovery so more weaknesses are found and exploited within a single testing cycle.

1. Entry occurs through AI-assisted interaction with live targets, where the model is given enough context to explore applications, log in, and attempt task progression.
2. Escalation comes from the agent’s ability to persist through failures, pivot to alternative paths, and continue probing until it finds a viable route.
3. Impact is faster vulnerability discovery and more effective exploitation of known weaknesses during real penetration-testing workflows.

NHI Mgmt Group analysis

AI-assisted offensive testing is becoming a governance problem, not just a tooling problem. Once a model is embedded in a workflow that can authenticate, navigate, and continue across multiple steps, it becomes a governed operational component. That changes the control conversation from model accuracy alone to scoping, logging, privilege, and task termination boundaries. Practitioners should treat these systems like other high-trust non-human workflows, with clear ownership and review.

Workflow performance matters more than benchmark performance. The article shows that the meaningful gains appear when the model is placed inside real penetration-testing sequences, not when it is judged in isolation. That is a reminder that security operations should evaluate AI systems on end-to-end task completion, failure handling, and escalation behaviour. The relevant question is whether the system makes decisions that align with operational intent.

Persistent access by AI systems creates a new control surface for identity teams. If an AI system can repeatedly attempt login, pivot, and continue after failure, then credential handling and access boundaries need to be explicit. This is where NHI governance intersects with agentic AI security: the model itself is not the identity, but the workflow it drives still depends on managed access and bounded authority. Teams should define the same lifecycle expectations they apply to other privileged non-human actors.

Named concept: agent workflow compression. This article shows how AI systems compress the time between discovery, access, and action by reducing the number of steps needed to reach a useful outcome. That shortens the window for human intervention and weakens assumptions built around slower, sequential security work. Practitioners should assume faster failure, faster success, and faster escalation all need to be governed together.

Security teams should not confuse offensive capability with safe autonomy. A model can be highly effective in a pentesting workflow and still require tight guardrails around where it runs, what it can touch, and when it must stop. The operational standard is not whether the model can act, but whether the surrounding control model keeps those actions accountable. The practitioner implication is to govern the system, not just the model.

What this signals

Agent workflow compression means the time between discovery, access, and action is shrinking inside AI-assisted security operations. That shortens the window for review and forces teams to think about execution limits, not just model quality.

For identity programmes, the more relevant question is whether AI tools that can authenticate into production-like environments are governed as privileged non-human actors. That means aligning task scope, credential boundaries, and auditability with the controls already used for high-risk service accounts and workload identities.

Security leaders should expect more AI systems to behave like operational intermediaries rather than passive assistants. When that happens, the surrounding governance model has to decide when the system may persist, when it must pivot, and when it must stop.

For practitioners

• Define agent execution boundaries Limit which systems AI-assisted security workflows can touch, what credentials they may use, and when they must stop. Use explicit allowlists for target environments and prohibit uncontrolled lateral movement during testing.
• Instrument login and retry behaviour Log every login attempt, failure, pivot, and retry so you can see whether the agent is wasting cycles or discovering a valid path. Separate failure handling from success paths in your telemetry.
• Treat source code access as amplification Assume that code visibility will improve AI-assisted discovery rather than simply validating what you already know. Prioritise high-value applications where white box context would materially increase the chance of exploitation.
• Set stop conditions for blocked paths Require the workflow to abort or escalate when credentials fail, bot detection appears, or the agent repeats the same path too often. That prevents over-persistence from becoming operational noise.
• Review privilege before production use If an AI system can authenticate into live environments, review whether the access it receives is task-scoped, time-bound, and monitored. Do not let testing convenience create standing access patterns.

Key takeaways

• AI-assisted security workflows now need governance that matches their ability to authenticate, pivot, and continue across real systems.
• XBOW’s testing shows a steep improvement curve, with GPT-5.5 reducing miss rate to 10% and changing what operational AI security teams should expect.
• The decisive control is no longer model output alone, but the combination of scope, retry policy, logging, and termination rules around the workflow.

Key terms

• Agent Workflow: An agent workflow is a machine-driven sequence where an AI system can call tools, access data, and continue execution with limited human involvement. These workflows need governance that sees API activity and tool use, not just browser sessions or user clicks.
• Miss Rate: Miss rate is the percentage of known issues a model or workflow fails to identify during testing. In offensive security evaluation, it is a more useful measure than isolated accuracy because it reflects how much vulnerability discovery is still being missed in practice.
• White Box Benchmark: A white box benchmark evaluates a system with source code or internal context available. It measures how well the model uses deeper visibility to discover issues, and it often exposes performance gaps that black box testing alone does not reveal.
• Task-Scoped Access: Task-scoped access is permission granted for one defined purpose and removed once the task is complete or the session expires. For non-human identities, it reduces standing privilege and limits how long an attacker can exploit a stolen credential.

What's in the full article

Xbow's full post covers the operational detail this post intentionally leaves for the source:

• Benchmark design details, including how vulnerable applications are frozen and replayed across model runs.
• Comparative workflow observations for GPT-5, Opus 4.6, and GPT-5.5 across black box and white box tasks.
• Logging-in and failure-handling behaviour that explains why the model feels faster in practice.
• How XBOW uses different models in different parts of its pentesting stack.

👉 The full Xbow post covers benchmark design, login behaviour, and model-by-model workflow comparisons.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is designed for practitioners who need to align identity controls with the operational realities of modern security workflows.