AI models are improving at offensive security workflows

At a glance

What this is: XBOW’s mid-year research reviews how several frontier AI models performed in offensive security workflows and shows that model capability is advancing quickly across cost tiers.

Why it matters: This matters because security teams now have to govern AI model use in testing, defense, and adversarial simulation without assuming yesterday’s guardrails will hold.

👉 Read Xbow's mid-year 2026 AI model security research report

Context

AI model security is no longer just about harmful outputs or prompt injection. It now includes whether frontier models can materially improve offensive security workflows, which changes how organisations think about model evaluation, containment, and oversight.

For IAM, PAM, and NHI programmes, the intersection matters because AI systems are increasingly embedded in security operations and testing pipelines. If models can independently shape offensive workflows, then access, tool use, and auditability around those models become governance problems, not just MLOps concerns.

Key questions

Q: How should security teams govern AI models that can call tools and access data?

A: Security teams should govern AI models as non-human identities with named owners, limited scope, short-lived credentials, and continuous authorization. The critical shift is to treat every tool call, data read, and update path as a privileged action that can be logged, revalidated, and revoked. Without that discipline, model risk becomes identity risk.

Q: Why do AI systems create NHI governance problems?

A: AI systems often rely on service accounts, tokens, APIs, and delegated permissions that behave like non-human identities. If those identities are not governed tightly, the system can access data or trigger actions beyond what people intended. That makes AI governance inseparable from identity and access control.

Q: What breaks when AI access is not scoped to the data the model actually needs?

A: Over-privilege turns AI into a high-speed data sprawl mechanism. The model can see, process, or expose information beyond its task, which increases the chance of leakage, poisoning, and compliance failure. The practical warning sign is when teams cannot explain why a given dataset is reachable by a given AI workflow.

Q: How do security teams know runtime AI guardrails are actually working?

A: Look for blocked poisoned inputs, flagged anomalous outputs, and traceable enforcement before responses reach users or downstream systems. If controls only inspect prompts or only inspect outputs, they leave a gap that attackers can exploit through manipulated data sources or tool responses.

Technical breakdown

How AI model capability translates into offensive workflow risk

In offensive security contexts, model capability is not just about answering questions well. It is about whether the model can support reconnaissance, triage, chaining of findings, and decision support fast enough to change attacker or tester behaviour. As models improve, they can compress the time between a weak signal and a useful action, which makes human oversight harder if it is added too late. The core issue is not the model alone, but the workflow around it: data retrieval, tool access, and escalation paths all expand the model's practical security impact.

Practical implication: Treat model evaluations as workflow-risk assessments, not simple accuracy checks.

Why offensive security benchmarking needs governance controls

Benchmarking a model for offensive security capability is different from benchmarking general reasoning. These tests should isolate whether the model can operate within bounded tasks, respect tool restrictions, and avoid leaking or amplifying sensitive context. Without that structure, teams can overestimate safety because a model appears constrained in a lab but behaves differently once connected to internal data or tools. The governance question is whether the model can be evaluated, monitored, and revoked like any other high-impact system with access to security-relevant assets.

Practical implication: Use controlled evaluation environments with explicit tool permissions and logging.

Model identity, access, and auditability in security operations

Once AI models are used in offensive or defensive security workflows, they behave like non-human actors with operational reach. That creates an identity problem: who granted the model access, what tools it can invoke, and how its actions are traced back to an accountable owner. This is where NHI governance becomes relevant even in an AI model security post, because model access often rides on service accounts, API keys, and delegated permissions. If those credentials are not scoped tightly, model capability becomes an access-control issue as much as a safety issue.

Practical implication: Bind model access to least-privilege identities and review every delegated tool path.

Threat narrative

Attacker objective: The objective is to use AI capability to improve the speed and effectiveness of offensive security operations while bypassing human bottlenecks.

1. Entry occurs when a security workflow connects an AI model to internal tools, data, or security telemetry without tight scoping or monitoring.
2. Escalation happens when the model can chain outputs into actions, enabling faster reconnaissance, prioritisation, or tool use than human operators expected.
3. Impact appears when that capability is used to accelerate offensive workflows, widen blast radius, or weaken confidence in existing containment assumptions.

NHI Mgmt Group analysis

Model capability is becoming a security control problem, not just an AI performance problem. When models can participate in offensive workflows, the question shifts from whether they are intelligent enough to whether they are governable enough. That makes access control, logging, and scope boundaries part of the model risk discussion. Security teams should treat model capability as something that must be bounded, not simply admired.

AI model security and NHI governance are converging around delegated access. Models do not act in a vacuum. They operate through service accounts, API keys, and tool permissions, which means AI model security inherits the same weaknesses that plague unmanaged non-human identities. The governance gap is often not the model itself but the identity layer beneath it. Practitioners should therefore evaluate model access as a privileged pathway.

Offensive security benchmarks create a new named risk we can call capability drift. As models improve across price tiers and update cycles, yesterday’s safe assumptions about what a model can or cannot do become stale quickly. That drift complicates approvals, red teaming thresholds, and operational guardrails. Security leaders should assume model capability profiles will move faster than policy refresh cycles.

Security teams need to separate model usefulness from model trust. A model can be valuable in testing and triage while still requiring strict containment. The mistake is assuming that productive use justifies broad tool access. That logic repeats the same error seen in identity governance, where convenience becomes entitlement. Practitioners should keep model trust conditional, reviewable, and revocable.

Offensive AI research is now an upstream signal for IAM and PAM teams. If models can compress reconnaissance or chaining, then the surrounding identity controls become the real enforcement layer. That means PAM, service account governance, and secrets management need to account for AI-assisted workflows, not just human operators. Teams should plan for model-mediated access paths as a standard threat model.

What this signals

Model capability is moving into the same governance zone as service accounts and machine credentials. That means security leaders should stop treating AI model evaluations as isolated lab exercises and start measuring how much real access each model path can reach in practice.

Capability drift: as frontier models improve, the policy assumptions built around earlier model behaviour decay quickly. This is a classic governance lag problem, and it now affects AI-assisted security workflows as much as it affects human access programmes. Teams should pair model evaluation cycles with access recertification and revocation testing.

Where models connect to internal tools, the operational risk starts to look like NHI risk in a new form. The most useful control lens is not just model safety but delegated access scope, especially where service accounts and API keys underwrite model actions. Review that boundary alongside the NIST SP 800-53 Rev 5 Security and Privacy Controls access and audit families.

For practitioners

• Define model access boundaries Limit each AI model to the minimum tools, scopes, and datasets needed for its task, and separate evaluation sandboxes from production security systems. Review delegated access the same way you would review a privileged service account.
• Log and review model actions Capture prompts, tool calls, retrieved context, and downstream actions so investigators can reconstruct what the model did and who authorised it. Without end-to-end logging, model-driven workflows become unauditable.
• Test against security-workflow abuse cases Red team models for tool misuse, chain-of-thought leakage, overbroad retrieval, and escalation through connected accounts before allowing them near sensitive environments.
• Tie model permissions to accountable owners Assign each model-integrated workflow a named business owner and a technical owner, then require periodic recertification of the access path, especially where service accounts or API keys are involved.

Key takeaways

• AI model security is now an access-governance issue because models can influence or execute offensive workflows through delegated tool paths.
• As model capability rises, assumptions about what is safe, bounded, or reviewable can decay faster than policy teams update controls.
• Security teams should govern model permissions, logging, and ownership with the same discipline they apply to privileged non-human identities.

Key terms

• Configuration Drift: Configuration drift is the gradual divergence between a system's intended secure state and the settings it actually runs with over time. In SaaS, drift often appears when admins change sharing, logging, or access controls under pressure and never return to validate the result.
• Model-Integrated Workflow: A process in which an AI model is connected to tools, data, or execution paths that affect real security work. The model may not make final decisions, but its outputs can trigger actions, which makes access scope, logging, and ownership critical control points.
• Delegated access path: A delegated access path is the chain of identities, tokens, connectors, and approvals that lets one system act through another. It becomes a governance concern when the path outlives the original approval or can be reused for actions beyond the intended business purpose.

What's in the full report

Xbow's full white paper covers the operational detail this post intentionally leaves for the source:

• Model-by-model evaluation results for GPT-5.5, Mythos Preview, Opus 4.7, GLM-5.2, Muse Spark 1.1, and Grok 4.5
• Cross-model themes that explain where capability differences matter in security workflows
• The implications section that maps model performance to security-team decision points
• The underlying white paper framing that teams can use for internal discussion and policy review

👉 Xbow's full white paper covers the model-by-model findings and security-team implications in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It gives practitioners a practical way to connect delegated access control with broader identity risk.