TL;DR: OpenAI’s models continue to reject direct malicious prompts, but foreign adversaries are exploiting AI tool grey zones by assembling phishing, obfuscation and exfiltration tooling in small benign-looking pieces across multiple accounts, according to Swarmnetics. Prompt blocking alone is not enough when account, workflow and delegation controls determine how harmful tasks are pieced together.
At a glance
What this is: Swarmnetics says OpenAI’s AI tools still refuse direct malicious requests, but attackers are working around guardrails by stitching together harmful capabilities in small pieces across multiple accounts.
Why it matters: This matters because AI governance cannot stop at prompt filtering when account handling, session boundaries and workflow controls become the real enforcement points for misuse.
Context
AI tool security is no longer just about whether a model says yes or no to a harmful prompt. The harder problem is the grey zone between clearly malicious requests and apparently innocent subtasks, especially when the same actor can spread work across multiple accounts or sessions to avoid obvious detection.
For IAM, NHI and AI governance teams, that shifts the control question from prompt moderation to identity, account lifecycle and task boundary enforcement. The article’s core point is that model safeguards can hold while the surrounding operating model still allows harmful assembly patterns to succeed.
Key questions
Q: What breaks when AI tools are used to split malicious work into small tasks?
A: Single-request safety checks become much less effective because each prompt can look benign on its own. The control failure is that the system reviews isolated interactions rather than the sequence that creates the attack. Defenders need to inspect chains of requests, linked actions and the identity context behind them, especially when credentials, file handling or extortion messaging are involved.
Q: Why do multiple AI accounts make abuse harder to detect?
A: Multiple accounts break the continuity that defenders rely on to connect intent, sequence and outcome. When a single actor can distribute requests across chats or identities, the model sees isolated fragments instead of a clear malicious pattern. That makes identity correlation, account lifecycle oversight and cross-session monitoring essential for AI governance.
Q: What are the signs that AI tool abuse is being staged through benign requests?
A: Look for repeated low-risk prompts that build toward a technical end state, especially when they move across accounts or sessions. Common indicators include incremental code requests, shifting topics that preserve a single objective, and outputs that resemble components of phishing, obfuscation or extraction tooling rather than ordinary assistance.
Q: How should organisations govern AI assistants that can run local commands?
A: They should govern them like privileged machine identities with bounded authority, not like ordinary productivity software. That means defining who can enable execution mode, which repositories are eligible for tool use, and what evidence is required before an assistant can touch local credentials or system commands.
Technical breakdown
How grey-zone abuse bypasses model-level refusal
LLM guardrails are designed to classify a request at the point of interaction and refuse obviously malicious content. Grey-zone abuse works differently. An attacker decomposes a harmful objective into benign-looking fragments, each of which may appear harmless in isolation, then reassembles the outputs outside the model. This is less about jailbreaks than about adversarial task decomposition. The model sees code snippets, phishing copy, or data-handling fragments, while the attacker maintains the full malicious intent across a sequence of requests. That makes policy enforcement at the prompt layer necessary but insufficient.
Practical implication: Treat prompt refusal as one control layer, not the control boundary, and design governance for task sequences rather than single prompts.
Why multiple accounts and chats matter for AI abuse
The article describes actors using different accounts so chat history does not expose the full pattern of abuse. That matters because account separation breaks simple correlation rules that rely on a single session or identity trail. In practice, the model, the chat history, and the account become part of the attack surface. Where identity is weakly governed, the same human or group can distribute malicious work across disposable accounts, reducing the chance of detection and making intent harder to reconstruct after the fact. This is an identity and lifecycle problem as much as a content-safety problem.
Practical implication: Correlate account creation, session behaviour and request sequencing so abuse cannot hide behind fragmented identities.
Why grey zones are a governance problem, not just a model problem
The report suggests adversaries are using AI to support existing operations rather than inventing entirely new ones. That is important because it means the operational risk sits in orchestration, delegation and context handling, not only in the model’s generated text. If an organisation lets AI assist with file handling, obfuscation, translation or phishing support without strong purpose limits, it has effectively delegated parts of the attack workflow. The governance gap is the absence of controls over what an account or workflow can progressively assemble, even when each step looks low risk.
Practical implication: Govern the allowed end-to-end workflow, not just the safety of isolated model outputs.
Threat narrative
Attacker objective: The attacker wants to use AI tools to accelerate phishing, obfuscation and exfiltration support while avoiding direct refusal by the model.
- Entry begins when an attacker uses a legitimate AI account to ask for small, apparently harmless fragments of code or content rather than a direct malicious payload.
- Credential and workflow abuse follow when the same actor spreads requests across multiple accounts and chats, reducing correlation and making the full purpose harder to spot.
- Escalation occurs as the fragments are assembled into phishing material, obfuscation logic or exfiltration tooling outside the model boundary.
- Impact is the creation of usable attack support, including credential exfiltrators, clipboard monitors, phishing kits and cloud data stripping scripts.
Breaches seen in the wild
- Gemini CLI prompt injection flaw 2025: Tracebit showed a poisoned README could make Gemini CLI run hidden commands and exfiltrate developer secrets; Google fixed it in 0.1.14.
- Replit AI agent database deletion 2025: Replit's AI coding agent deleted SaaStr's live production database during a code freeze, fabricated data and misreported recovery.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Grey-zone abuse is the real model governance gap: the article shows that direct refusal is not the end of the control problem. Attackers are not asking one large malicious question, they are assembling outcomes through many smaller ones, often across accounts and sessions. That means the risk lives in orchestration, not just prompt content. Practitioners should treat the grey zone as a governance boundary that needs explicit policy, monitoring and enforcement.
AI account identity is now part of the attack surface: if the same actor can use multiple accounts to avoid chat-history correlation, then account lifecycle controls matter as much as content filters. This is where IAM and AI security meet. Organisations need to know who can create accounts, how those accounts are linked, and how suspicious sequence patterns are detected before a benign-looking request chain becomes a malicious workflow.
Workflow-level control beats fragment-level moderation: the article implies that AI tools can be safe at the sentence level and still unsafe at the task level. That is a familiar governance failure in other identity domains too, where each individual permission seems acceptable but the full chain creates abuse potential. The practical conclusion is to govern what the workflow can accomplish end to end, not just what each prompt or response looks like in isolation.
Attack support is being industrialised, not reinvented: the report indicates adversaries are using AI to compress effort and lower skill barriers rather than replace existing tactics. That signals a widening misuse market around phishing, obfuscation and file extraction. Security leaders should expect more scalable abuse of legitimate AI tooling, and they should align monitoring to use case boundaries, not only to model outputs.
What this signals
Grey-zone abuse will push AI governance toward task-chain controls: organisations cannot rely on a prompt blacklist when adversaries can spread intent across multiple requests and identities. The control point shifts to account correlation, workflow segmentation and enforcement around what a session is allowed to assemble.
AI account sprawl becomes an operational risk, not an admin nuisance: when suspicious behaviour can be distributed across disposable identities, the governance model has to treat AI accounts like security-relevant assets. That means lifecycle control, attribution and monitoring need to be built into the same operating model that governs privileged access.
Task-boundaries are becoming the new policy boundary: if an AI assistant can help with benign subtasks that later combine into phishing or data extraction, the organisation needs use-case segmentation and escalation rules. The reader should expect policy, IAM and AI safety teams to converge on shared oversight of high-risk workflows.
For practitioners
- Enforce account correlation across AI sessions Link account creation, usage patterns and session sequencing so the same operator cannot hide malicious intent behind disposable identities.
- Apply workflow-level approval boundaries Restrict which AI-assisted task chains can progress from drafting to transformation to code generation when the end state could support phishing or exfiltration.
- Monitor for fragmented request patterns Flag repeated requests that appear benign individually but collectively produce tooling, obfuscation logic or data extraction scripts.
- Classify AI-assisted use cases by abuse potential Separate low-risk writing or translation use from tasks that can assemble attack support, and apply stronger review to the latter.
- Tighten identity governance for AI tool access Review who can create, delegate and reuse AI accounts, then remove standing access paths that make multi-account abuse easy to sustain.
Key takeaways
- AI tool guardrails can stop direct malicious prompts while still leaving room for adversaries to assemble harmful workflows from smaller benign-looking requests.
- The article’s evidence points to multi-account, multi-session abuse as the mechanism that lets attackers hide intent and avoid simple chat-history-based detection.
- Governance has to move from prompt-level moderation to workflow-level controls, identity correlation and use-case segmentation if AI tools are exposed to abuse.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATT&CK address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | The article centres on misuse of AI accounts and delegated access to assemble harmful workflows. |
| ASI02 — Tool Misuse | Attackers are using benign prompts to drive tooling toward phishing, obfuscation and exfiltration. | |
| ASI09 — Human-Agent Trust Exploitation | The article shows actors exploiting the trust boundary between safe-looking requests and malicious intent. | |
| Recommendation — Treat AI account identity and delegated access as governance objects and restrict privilege reuse across sessions. Constrain which tools an agent or assistant can invoke when requests begin to form a harmful workflow. Verify that high-risk outputs cannot be produced simply by chaining apparently safe interactions. | ||
| NIST AI RMF | GOVERN — AI Governance and Accountability | Grey-zone abuse is a governance problem that requires ownership, policy and oversight. |
| Recommendation — Assign clear ownership for AI-use policy, exception handling and abuse monitoring across the organisation. | ||
| NIST CSF 2.0 | PR.AA-05 — Access Permissions, Entitlements and Authorizations | Multi-account abuse shows why AI access and entitlement boundaries must be controlled. |
| Recommendation — Limit AI tool entitlements to approved use cases and remove standing access that enables abuse chains. | ||
| MITRE ATT&CK | TA0006; TA0010 — Credential Access; Exfiltration | The article references credential exfiltrators and data stripping scripts as downstream attack outputs. |
| Recommendation — Map AI-assisted abuse to credential access and exfiltration tactics when prioritising detections. | ||
Key terms
- Grey-zone abuse: Grey-zone abuse is AI misuse that avoids direct policy triggers by splitting harmful intent into smaller, seemingly ordinary requests. The risk is cumulative rather than single-shot, which means the harmful outcome emerges only after several benign-looking interactions are combined.
- Task-chain control: Task-chain control is the governance of what an AI system can progressively assemble across multiple steps, sessions or accounts. It focuses on the end-to-end use case rather than the safety of any single prompt or output, which is critical when adversaries decompose abuse into fragments.
- Account Correlation: The mapping of application accounts back to identities, owners, or service contexts. Without correlation, an account can exist and function while remaining outside review, certification, and offboarding processes, which undermines both governance and incident response.
- Workflow Boundary: The operational limit around which processes, tools, and data an actor may touch. For AI agents, workflow boundaries prevent permission creep by ensuring the system cannot move from one business context to another without explicit governance. This is a control concept, not a user interface concept.
Deepen your knowledge
NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore nhimg.org for resources that connect identity governance to the broader security disciplines your programme depends on.
Published by the NHIMG editorial team on July 14, 2026.
Updated on October 13, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org