AI tool guardrails still hold, but attackers exploit grey zones

At a glance

What this is: OpenAI’s threat report says direct malicious requests are usually blocked, but sophisticated adversaries are assembling attack capabilities through fragmented, low-signal interactions.

Why it matters: That matters because AI governance cannot rely on prompt filtering alone when account reuse, delegated access, and workflow stitching let threat actors move around model guardrails.

By the numbers:

• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes

👉 Read Swarmnetics’ analysis of OpenAI’s latest threat report on AI misuse patterns

Context

AI tool abuse is increasingly a governance problem, not just a model-safety problem. The core issue is that safeguards can block overtly malicious prompts while still leaving room for attackers to assemble harmful capability through small, ordinary-looking requests, repeated sessions, and multiple accounts. For identity and security teams, that makes account provenance, session continuity, and delegated access part of the control surface.

The article also sits at the intersection of AI governance and non-human identity security because the abuse pattern depends on fragmented identities, credentialed access, and operational stitching across tools. That is a familiar risk shape in NHI programmes: attackers do not need one dramatic request if they can accumulate enough benign-looking interactions to build a working attack chain.

Key questions

Q: How should security teams govern AI workflows that use multiple tools and data sources?

A: Security teams should govern AI workflows by placing explicit authorization at each decision point, not by relying on the permissions attached to the surrounding application or service account. The practical goal is to scope read, retrieve, and execute access separately so the workflow cannot inherit broader reach than it needs for the task.

Q: Why do AI tools create a different abuse pattern than traditional application misuse?

A: AI tools can help an attacker assemble harmful output incrementally, even when each individual request looks harmless. That shifts the risk from single-event abuse to staged workflow abuse, where translation, debugging, and code polishing become part of the attack path. Traditional controls built around one request or one session often miss that distributed pattern.

Q: What do security teams get wrong about prompt filtering for AI agents?

A: They treat prompt filtering as if it were a complete control layer. It is only a first screen against obvious malicious text. Indirect prompt injection often hides in legitimate-looking documents or tickets, and the agent can still follow the hidden instruction through approved connectors and valid credentials.

Q: Who is accountable when AI tools are abused to support malware operations?

A: Accountability sits across AI governance, security operations, and identity ownership. Teams that approve models, expose them to users, or connect them to tools need documented controls for abuse detection, access restriction, and incident response. Where AI assistants are integrated into workflows, governance must cover both the model and the permissions around it.

Technical breakdown

How grey-zone prompting bypasses direct refusal controls

Large language models often apply policy filters at the request boundary, which is why direct malicious prompts are rejected more reliably than they were in earlier systems. The grey-zone problem appears when the attacker decomposes a harmful objective into smaller tasks that look individually benign, such as debugging, obfuscation, translation, or file handling. Each step may pass moderation, yet the combined output still supports phishing, credential theft, or malware assembly. Multiple accounts further reduce continuity, making it harder for a single conversation history to reveal intent. The failure mode is not model intelligence alone, but the inability of controls to understand multi-step intent across sessions and identities.

Practical implication: security teams need cross-session intent review and account-level correlation, not just prompt filters.

Why account fragmentation changes the AI abuse model

When an adversary uses several accounts, each prompt becomes less informative on its own, and the platform’s visibility into the full workflow drops sharply. This matters because modern AI governance often treats the user session as the unit of control, while the attacker treats the task chain as the unit of work. In practice, this can turn one user objective into a distributed sequence of benign interactions that evade both moderation and anomaly detection. The identity layer, including account lifecycle, attribution, and access provenance, becomes central to how AI abuse is detected and investigated.

Practical implication: correlate AI activity by identity, device, and workflow lineage rather than by prompt alone.

How AI is lowering the barrier to attack assembly

The report’s most important technical signal is not that AI is inventing novel attack classes, but that it is compressing the effort required to assemble existing ones. The model can help with translation, code polishing, obfuscation, and incremental script generation, which means less experienced actors can execute campaigns that previously required more coordination. That is why the security problem extends beyond content moderation into operational misuse of AI as an attack acceleration layer. For defenders, the key question is whether the organisation can see, classify, and constrain AI-assisted workflows before they mature into a usable intrusion path.

Practical implication: treat AI-assisted task chaining as a misuse pattern and monitor for incremental code generation, translation, and evasion workflows.

Threat narrative

Attacker objective: The objective is to use AI systems as a force multiplier for phishing, evasion, and attack-tool assembly while avoiding direct refusal controls.

1. Entry occurs through ordinary-looking AI interactions spread across multiple accounts, each request framed as a small and apparently benign task.
2. Escalation happens when the attacker stitches those outputs together into phishing content, obfuscation logic, credential exfiltration tooling, or cloud data stripping scripts.
3. Impact is the accelerated production of attack tooling and scam content with less skill, less coordination, and lower cost than a manual operation would require.

Breaches seen in the wild

• MITRE ATT&CK Enterprise Matrix — MITRE ATT&CK Enterprise — adversary tactics and techniques, threat detection, attack chain mapping, credential access, lateral movement, privilege escalation.
• Replit AI Tool Database Deletion — Replit vibe coding AI assistant deletes live production database and creates 4,000 fake user records.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Grey-zone abuse is becoming the dominant AI security problem: the most consequential misuse is no longer direct jailbreak success, but incremental task decomposition that evades single-request controls. That pattern collapses the assumption that moderation at the prompt layer is enough. For AI governance, the control boundary has to move to identity, workflow lineage, and stateful oversight across sessions.

AI account fragmentation is an identity governance issue, not just a content-safety issue: when attackers use multiple accounts to hide intent, the platform loses the ability to connect benign-looking prompts into a malicious chain. That is the same structural problem identity teams face with distributed non-human activity, where attribution, lifecycle, and session continuity determine whether abuse is visible. Practitioners should treat account provenance as part of AI risk governance.

Model refusal rates do not equal operational safety: a system can reject overtly malicious requests and still materially aid attack preparation through translation, debugging, obfuscation, and staged generation. That distinction matters because many governance programmes still measure success by refusal outcomes rather than by whether harmful workflows can be assembled end to end. Security teams should evaluate the full attack workflow, not isolated prompt outcomes.

AI governance is converging with NHI governance: as AI tools become part of operational work, they behave less like passive applications and more like credentialed systems participating in business processes. That creates a named concept we should keep in view: workflow stitching risk, meaning the ability to turn separated, low-risk interactions into a complete harmful chain. The practical conclusion is that identity, authorisation, and telemetry must be designed for stitched behaviour, not single-turn misuse.

From our research:

• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes, according to LLMjacking: How Attackers Hijack AI Using Compromised NHIs.
• Our research also found that the average estimated time to remediate a leaked secret is 27 days, even though 75% of organisations express strong confidence in their secrets management capabilities.
• For adjacent identity risk, see The State of Secrets in AppSec for how secret sprawl and delayed remediation shape attack windows.

What this signals

AI abuse is moving from obvious misuse to chained misuse, which means detection has to shift from individual prompts to cross-session behaviour. That is where identity telemetry matters most, because the security team needs to know whether a benign prompt is part of a larger delegated workflow. The operational signal is less about what the model said and more about how the account behaved over time.

Workflow stitching risk: a model can be safe at the request boundary and still unsafe at the workflow boundary. That distinction should change how teams design logging, abuse response, and privilege constraints for AI access, especially when accounts are shared, rotating, or loosely attributed. A stateful control model is now more useful than one built around isolated prompt moderation.

For practitioners, the near-term priority is to align AI governance with NHI governance and session analytics. That means reviewing access ownership, shrinking the number of accounts that can generate operational output, and using established identity controls to spot when AI becomes part of an attack chain rather than a productivity layer.

For practitioners

• Map AI abuse paths by identity and workflow Correlate prompts, sessions, devices, and account histories so that multiple low-risk interactions can be investigated as one chained activity. Focus on role changes, account reuse, and repeated requests that gradually converge on code generation, obfuscation, or exfiltration.
• Tighten lifecycle controls for AI access accounts Apply joiner-mover-leaver discipline to AI users and service identities, including provenance checks, access review, and rapid deprovisioning when behaviour changes. Multi-account abuse is easier when identity ownership is vague or stale.
• Add workflow-level detections for staged abuse Build detections for translation bursts, incremental script refinement, repeated debugging, and evasive request sequencing because these are often the precursors to phishing kits, credential tooling, and cloud data extractors.
• Treat AI tool output as a control point Inspect generated code, messages, and transformation outputs before they are reused downstream, especially where the output can become part of phishing, obfuscation, or file extraction workflows.

Key takeaways

• Direct malicious prompts are not the main failure mode. The real risk is that attackers can assemble harmful capability in small, low-signal pieces across multiple accounts.
• AI abuse is now an identity and workflow problem as much as a model-safety problem. Without account correlation, intent can disappear inside ordinary-looking interactions.
• Security teams should govern AI access like other high-risk operational systems, with lifecycle control, session visibility, and downstream output review.

Key terms

• Grey-zone abuse: Grey-zone abuse is AI misuse that avoids direct policy triggers by splitting harmful intent into smaller, seemingly ordinary requests. The risk is cumulative rather than single-shot, which means the harmful outcome emerges only after several benign-looking interactions are combined.
• Workflow stitching risk: Workflow stitching risk is the ability to combine separate AI interactions into one harmful operational chain. It matters because security controls that evaluate requests one by one can miss the full attack path when an adversary uses multiple sessions or accounts to hide intent.
• Cross-Session Correlation: Cross-session correlation is the practice of linking events across accounts, devices, infrastructure, and time to reveal coordinated behaviour. It is the difference between seeing one clean session and seeing an abuse campaign that only becomes visible when many clean sessions are analyzed together.
• AI account provenance: AI account provenance is the ability to trace who created, owns, and used an AI account over time. It supports attribution, abuse investigation, and access governance, especially when multiple accounts are used to break up malicious activity into low-signal pieces.

What's in the full analysis

Swarmnetics' full analysis covers the operational detail this post intentionally leaves for the source:

• Examples of the model guardrails OpenAI says are holding up against direct malicious requests.
• The specific nation-state and criminal activity patterns the report links to grey-zone abuse and staged prompting.
• The report’s own evidence on phishing, scam generation, and multi-account coordination across languages.
• Additional context on how OpenAI differentiates direct misuse from incremental tool assembly.

👉 Swarmnetics’ full post covers the source evidence, actor examples, and the reported AI abuse patterns in more detail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, IAM, secrets management, and agentic AI identity. It helps practitioners connect identity controls to the broader security programmes their organisations rely on.