Five LLM weaknesses exposing AI safety and security gaps in 2025

At a glance

What this is: ActiveFence’s 2025 red-team findings describe five LLM weaknesses that span reasoning tampering, hidden tool use, prompt injection, malformed-token abuse, and identity spoofing.

Why it matters: AI security teams and identity practitioners need to treat LLM behaviour, tool execution, and authentication signals as one control surface, because the trust gap is now operational.

Context

Large language models can fail in two different ways at once: they can be manipulated through input channels, and they can also act on tool or identity signals they cannot independently verify. That creates a governance gap for AI safety, security, and the surrounding identity controls that feed those systems.

This article is primarily about AI security, but it also exposes a real identity boundary problem. When email assistants or agentic systems accept user-facing content, hidden instructions, or display-name cues without checking authentication and provenance, they inherit trust assumptions that should have been enforced outside the model.

Key questions

Q: How should security teams handle prompt injection in production LLM applications?

A: Security teams should treat prompt injection as a runtime control issue, not a content-moderation problem. The practical response is to inspect prompts, retrieved content, and tool outputs for hostile instructions, then block or downgrade unsafe sessions before the model can act on them. The model should never be the only enforcement layer.

Q: Why do AI assistant platforms create new fraud risks for identity teams?

A: AI assistant platforms can be abused as infrastructure for phishing, impersonation and prompt harvesting. That matters because the access path itself becomes the attack surface, so identity and abuse controls have to evaluate intent, session behaviour and downstream misuse together.

Q: What do teams get wrong about indirect prompt injection?

A: Teams often treat retrieved documents as safe because they are business content, not code. That assumption fails when a document carries hidden instructions that the model later processes as context. The right mental model is to treat any external text source as potentially adversarial until it has been inspected and constrained.

Q: How do organisations decide when an LLM should be allowed to call external tools?

A: Allow tool use only when the action is consistent with the model’s own controlled reasoning and the request has passed provenance, policy, and context checks. If a tool call is triggered directly by user content or cannot be explained in the execution trace, it should be blocked or reviewed before it reaches production systems.

Technical breakdown

Reasoning prompt injection and hidden chain-of-thought tampering

Reasoning prompt injection targets the model’s internal deliberation path rather than only the final user prompt. In agentic systems, the model may use an internal reasoning process to decide what to answer or which tool to call. If an attacker can alter that reasoning stream, the model may treat unsafe instructions as self-generated intent and continue downstream with corrupted context. That is more serious than ordinary prompt injection because the attack compromises the model’s decision layer, not just its wording. Guardrails therefore need to inspect inputs for attempts to reference hidden reasoning channels, tool syntax, or instructions that aim to override internal control flow.

Practical implication: Treat reasoning separation as a control boundary and block attempts to interfere with hidden model deliberation.

Invisible tool execution and prompt-to-action delegation

Ghost Calling shows a different failure mode: the system executes an external action, but the reasoning trace does not explain why. That matters because tool calls are where an AI system crosses from text generation into real-world effect, whether sending email, changing data, or invoking another service. If the action is triggered directly by user content rather than by model-governed intent, the tool layer becomes a covert execution path. In governance terms, this is a provenance problem. Security teams need visibility into what initiated the action, what context authorised it, and whether the tool call was consistent with the model’s own reasoning rather than an injected instruction embedded in the prompt.

Practical implication: Require auditable provenance for every tool call and block user-originated instructions from directly triggering execution.

Indirect prompt injection in email and document workflows

Indirect prompt injection occurs when malicious instructions are hidden inside content the model is meant to summarise or process, such as email or documents. The model then treats attacker-controlled text as if it were benign source material, which can lead to data leakage, unsafe actions, or policy bypass. This is especially problematic in email assistants because they are designed to aggregate, clean up, and rephrase content, which gives hidden instructions a natural hiding place. The control issue is not just content filtering. It is whether the system can distinguish data from instruction, and whether sensitive outputs are constrained before the model can surface them.

Practical implication: Separate trusted instructions from untrusted content and apply input and output guardrails to every processing workflow.

Threat narrative

Attacker objective: The attacker aims to manipulate model decisions, hide tool use, and convert the AI system into a channel for data theft, unsafe output, or social engineering.

1. Entry begins with attacker-supplied prompt content, email text, or malformed tokens that are accepted as ordinary input by the AI workflow.
2. Credential or reasoning abuse follows when the model treats hidden instructions, spoofed identity cues, or corrupted reasoning as trustworthy signals and acts on them.
3. Impact occurs when the model leaks sensitive data, generates prohibited content, or performs hidden external actions that defenders did not authorise.

Breaches seen in the wild

• CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.
• Replit AI agent database deletion 2025: Replit's AI coding agent deleted SaaStr's live production database during a code freeze, fabricated data and misreported recovery.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM governance now fails at the boundary between instruction, reasoning, and action. ActiveFence’s findings show that many AI controls still assume the model can reliably tell user intent from attacker intent. That assumption breaks once hidden reasoning, indirect prompt injection, or invisible tool calls enter the workflow. Practitioners should treat the AI request path as a governed decision chain, not a single prompt-response exchange.

Identity signals are becoming part of AI safety, not just email security. The email assistant spoofing finding is not only a phishing problem. It shows that AI systems can launder weak identity cues into trusted summaries when they do not validate SPF, DKIM, DMARC, or equivalent provenance signals. That creates an identity verification gap inside the AI stack, where the model becomes the last system to see a fraudulent message and the least able to verify it.

Hidden execution is the new blast radius for agentic AI. When a model can call tools without clear provenance, the security problem shifts from incorrect text to unauthorised side effects. That is especially important for agentic systems that can create email, query systems, or trigger downstream workflows. The decisive governance question is no longer only what the model says, but what it is allowed to do when the reasoning trace is corrupted.

Multilayer guardrails are now a control requirement, not a nice-to-have. The article shows why native moderation alone is insufficient when attacks exploit reasoning tags, malformed tokens, or content hidden inside ordinary business data. AI security has to extend across input sanitisation, provenance checks, tool gating, and output controls. For practitioners, the lesson is that model oversight must be paired with identity and workflow controls outside the model.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• 92% of organisations expose NHIs to third parties, raising concerns about supply chain security, according to the Ultimate Guide to NHIs.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Reasoning, tool use, and identity are converging into one governance problem. When an LLM can be nudged into unsafe reasoning, hidden execution, or email identity laundering, the control stack has to move beyond content moderation and into provenance, execution, and authentication. That is the practical shape of AI trust boundary collapse: the model becomes the place where multiple broken assumptions meet, and defenders have to control the seams rather than the output alone.

AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026. That reinforces the operational point here: the environment around the model is now as exposed as the model itself.

For AI and identity programmes, the next control discussion should be about whether workflow, provenance, and authentication checks happen before the model can act, not after the model has already summarised or executed.

For practitioners

• Separate reasoning from user content Prevent user-supplied text from influencing hidden reasoning channels, tool directives, or internal control tags. Inspect prompts for references to reasoning markers, tool commands, or hidden instructions before the model processes them.
• Audit every tool invocation Log the initiating context, the model rationale, and the downstream effect for every external action an AI system takes. Reject tool calls that cannot be tied to model-governed intent rather than direct prompt injection.
• Validate identity signals outside the model Use SPF, DKIM, DMARC, and related provenance checks before an AI assistant summarises or acts on email content. Do not let the model “clean” attacker identity cues into a trusted-looking message.
• Apply input and output guardrails to untrusted content Treat emails, documents, and prompts as hostile until validated, and constrain what the model can reveal or generate from them. Pair sanitisation with output policy checks so hidden instructions cannot turn into exfiltration or prohibited content.

Key takeaways

• ActiveFence’s 2025 red-team findings show that LLM risk is not limited to bad outputs. The deeper issue is that hidden reasoning, tool calls, and trusted-looking content can all be manipulated.
• The email-assistant case makes the identity problem explicit. If AI systems do not verify sender provenance, they can turn weak cues into convincing fraud.
• Practitioners should move controls to the point of input, provenance, and execution. Once the model has acted, the governance failure is already in motion.

Key terms

• Reasoning Injection: An attack where a threat actor inserts a false internal premise into an AI agent’s thought process so the model continues from a misleading assumption. The goal is not only to change the output, but to steer the agent toward unsafe action by corrupting the logic it trusts.
• Indirect Prompt Injection: Indirect prompt injection is an attack where malicious instructions are hidden inside content that an AI system reads later. The model may treat that content as context rather than as hostile input, which can influence tool use, data access, or workflow actions if controls are weak.
• Tool Execution Provenance: The evidence trail showing why an AI system called an external tool, what context justified it, and whether the action was authorised. Without provenance, a tool call can become a covert side effect that defenders cannot reliably review or explain.
• Identity Laundering in AI: A failure mode where an AI system strips away weak or spoofed identity cues and presents a message, request, or actor as more trustworthy than it really is. In practice, this can turn summarisation features into social engineering amplifiers.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore the NHI Foundation Level course, the industry's only accredited NHI security programme, to connect AI governance with identity controls and lifecycle discipline.