Adversarial AI exposes an identity gap in security operations

At a glance

What this is: This is an analysis of adversarial AI that separates attacks on AI systems from AI-enabled attacks against defenders, with a clear emphasis on identity, logging, and detection gaps.

Why it matters: It matters because AI features, AI agents, and AI credentials expand the attack surface in ways that existing IAM, SIEM, and SOC processes often do not cover well enough.

By the numbers:

• 28% of breaches are initiated through phishing or social engineering.
• AI-enhanced phishing can boost attack profitability up to 50x.
• AI-related vulnerabilities are the fastest-growing cyber risk in 2026, named as the top concern by 87% of security leaders.

👉 Read Panther's analysis of adversarial AI attacks, risks, and defenses

Context

Adversarial AI now stretches across both offense and defense, which is why standard SOC and IAM assumptions break down quickly once AI systems start handling real data, tools, and credentials. In this article, Panther frames the problem as two-sided: securing AI systems themselves and defending against AI-enabled attacker behaviour, with identity and logging as the first control layer.

That framing is relevant for IAM and NHI programmes because LLMs, AI agents, and service accounts increasingly share the same access paths as production workloads. Where AI services authenticate with keys, roles, or tokens, the boundary between model risk and identity risk disappears, and governance has to treat AI systems as privileged actors rather than isolated features.

Key questions

Q: How should security teams govern AI agents that call internal tools from Kubernetes?

A: They should separate runtime containment from identity governance. Kubernetes can isolate execution, but the tool boundary still needs delegated identity, per-request authorization, and action-level logging. Without those controls, the agent behaves like an unscoped service account with model-driven output, which makes attribution and revocation much harder.

Q: Why do exposed AI secrets create more risk than ordinary cloud credentials?

A: AI secrets often sit between data sources, model access, and downstream business systems, so a single key can expose prompts, outputs, and linked datasets while also enabling access to adjacent services. The result is a larger blast radius. Teams should assess not only whether a secret exists, but what systems it can reach.

Q: What do security teams get wrong about prompt injection defence?

A: They often assume better blocklists will solve the problem, but obfuscation simply changes the shape of the payload. Real defence requires examining meaning across the full interaction, including retrieved content and model responses. If the control cannot interpret intent, it will keep missing the attack class it is meant to stop.

Q: Which controls matter most when AI services are targeted for abuse?

A: Start with least privilege, secrets rotation, and centralized logging before adding model guardrails. That order matters because guardrails cannot explain who used a credential or what scope it had. If the identity layer is weak, the AI layer becomes much harder to investigate or contain.

Technical breakdown

Prompt injection and indirect instruction abuse

Prompt injection is the manipulation of an LLM through malicious instructions hidden in user input or external content such as documents, emails, or tickets. Indirect prompt injection is more dangerous because the payload sits in content the model processes without a user intentionally supplying the malicious text. The model follows the highest-priority instruction it can infer, which can cause data leakage, unsafe actions, or delegation to downstream tools. In agentic workflows, the risk expands because the model can pass compromised instructions into search, ticketing, or code execution steps.

Practical implication: isolate system prompts, normalise inputs, and inspect external content before it reaches model or agent workflows.

LLMjacking and cloud credential abuse

LLMjacking is the theft and misuse of cloud or API credentials to access LLM services on the victim's account. The attacker does not need to break the model itself if they can obtain keys, roles, or tokens that authorize inference calls. Once inside, they can consume compute, generate cost, and sometimes pivot into broader cloud resources tied to the same identity. This is fundamentally an IAM and secrets-management issue disguised as AI misuse, because the attack path depends on exposed, reusable credentials rather than model weakness alone.

Practical implication: scope AI service identities tightly, remove hardcoded keys, and monitor token use and caller identity for abuse.

Telemetry, detection rules, and model-layer guardrails

Defending adversarial AI requires three layers working together. Telemetry gives you caller identity, model name, token count, source IP, and status codes so you can spot abuse patterns. Detection rules then identify anomalous API consumption, prompt injection indicators, AI agent lateral movement, and credential access anomalies. Guardrails at the model layer reduce unsafe inputs and outputs, but they do not replace identity or logging controls. Without centralized logs, model safety features cannot tell you who invoked what, from where, and with which permissions.

Practical implication: centralize AI logs first, then add AI-specific detections and guardrails as a second control plane.

Threat narrative

Attacker objective: The attacker wants to turn AI access into operational leverage, whether that means stealing compute, exfiltrating data, or using compromised identities to expand footholds across the cloud stack.

1. Entry occurs when attackers use phishing, exposed credentials, or vulnerable cloud services to reach AI systems or the identities behind them.
2. Escalation follows when stolen API keys, service accounts, or delegated roles are reused to call LLM services, fetch metadata credentials, or move laterally into adjacent cloud resources.
3. Impact is cost abuse, data exposure, malware control, or corrupted AI-assisted decision-making that extends beyond the original AI workload.

NHI Mgmt Group analysis

Adversarial AI is now an identity problem, not only a model problem. The article correctly separates attacks on AI systems from AI-enabled attacker tradecraft, but the governance failure is that both still depend on identity, access, and telemetry. When an AI agent can call tools, query services, or inherit privileges, it becomes another non-human identity to govern. That means IAM, PAM, and NHI controls must extend into AI operations, not sit beside them.

LLMjacking is the clearest example of credential governance failure in AI operations. Attackers do not need model internals when exposed keys, roles, or tokens give them a direct path into paid inference services and adjacent cloud resources. The article's own remediation advice points in the right direction because least privilege, secrets management, and identity logging are the real controls. Practitioner conclusion: AI security starts with NHI governance for the services that invoke AI.

Prompt injection exposes a control boundary that many teams still misunderstand. Treating model inputs as trusted text is a governance error, especially when those inputs come from email, Jira, shared documents, or other systems that already sit inside enterprise workflows. The named concept here is instruction trust gap, the failure to separate content ingestion from authoritative control instructions. Practitioner conclusion: any AI workflow that can act on external text needs explicit trust zoning.

Centralized telemetry is the difference between detecting AI abuse and merely observing it after the fact. Logs that remain split across cloud consoles and AI platforms make it hard to correlate caller identity, token consumption, and downstream tool use. That is why AI detection belongs in the same operational model as SOC telemetry, but with identity context added. Practitioner conclusion: if you cannot trace an AI action back to a service identity, you do not have governable AI.

What this signals

Instruction trust gap: the real operational risk in adversarial AI is the gap between what a model reads and what a system is authorised to do. Teams that cannot separate retrieved content, user input, and privileged instructions will struggle to govern agentic workflows safely, especially when those workflows can touch tickets, code, or cloud resources.

The next programme shift is toward identity-centric AI security, where service accounts, API keys, and agent permissions are treated as governable assets rather than implementation details. That aligns with broader NHI practice and with frameworks such as the NIST Cybersecurity Framework 2.0 and the MITRE ATT&CK Enterprise Matrix, both of which reward traceability and control coverage over assumption-driven trust.

For practitioners

• Define AI service identities as governed workloads Assign each AI application, agent, or pipeline its own service identity, scope permissions to the minimum model ARNs or API actions required, and remove shared keys wherever possible. Treat every AI caller as a separately reviewable identity boundary, not as a generic application integration.
• Centralize AI telemetry with identity context Collect caller identity, model name, token counts, source IP, and response status for every AI request, then route those logs into the SIEM with the same retention and correlation rules used for other privileged workloads.
• Write detections for AI-specific abuse patterns Create rules for anomalous LLM token consumption, prompt injection indicators, credential access on AI service accounts, and suspicious AI agent calls to IAM or metadata services. Test them against benign automation so they distinguish normal workload behaviour from abuse.
• Separate external content from control instructions Normalize Unicode, scan documents and messages before model ingestion, and keep system instructions structurally separate from user or retrieved content. This reduces indirect prompt injection risk in RAG and agent workflows that process tickets, emails, or shared files.
• Apply model-layer guardrails to production endpoints Enable content filtering, topic denial, and PII redaction on all production AI endpoints, but treat these as last-line controls. Pair them with rate limits and secrets governance so the model layer cannot compensate for weak identity boundaries.

Key takeaways

• Adversarial AI creates two separate risk channels, one against the model and one through AI-enabled attacker tradecraft.
• Identity hardening and centralized telemetry are the practical foundations for detecting LLMjacking, prompt injection, and AI agent misuse.
• Security teams should govern AI services as privileged non-human identities before model-layer controls can be trusted.

Key terms

• Adversarial AI: Adversarial AI refers to AI used by attackers to scale reconnaissance, impersonation, or abuse in ways that overwhelm normal manual review. For defenders, the issue is not just malicious model use, but the way machine-speed behaviour changes the timing, volume, and accuracy demands on identity and fraud controls.
• LLMjacking: Abuse of cloud AI services through stolen machine credentials rather than human user accounts. The attacker uses valid non-human identities such as API keys or tokens to enumerate model access, invoke endpoints, and create cost, data, or policy exposure under the victim's tenancy.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Non-Human Identity (NHI): A digital identity assigned to a non-human entity such as a software application, service account, API key, bot, machine, or AI agent that enables it to authenticate and interact with systems without direct human involvement. NHIs now outnumber human identities in most enterprises by 25 to 50 times.

What's in the full article

Panther's full blog covers the operational detail this post intentionally leaves for the source:

• Step-by-step control examples for AWS, Azure, and GCP AI services that practitioners can adapt to their own cloud estate.
• Detection logic ideas for prompt injection, anomalous token usage, and AI agent calls into IAM or metadata services.
• Concrete logging fields and schema guidance for building AI telemetry that analysts can query in the SIEM.
• Defensive sequencing for lean teams that need to choose between identity hardening, logging, and model guardrails.

👉 Panther's full blog adds implementation detail on AI logging, detection rules, and identity hardening.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management for practitioners who need to manage privileged automation safely. It gives security teams a shared control language for identity, access, and lifecycle governance across modern environments.