AI security now spans model corruption and runtime abuse

At a glance

What this is: This is an analysis of how AI security risk now spans both model integrity and runtime application abuse, with the central finding that static defences fail against dynamic, agentic behaviour.

Why it matters: It matters because IAM, NHI and AI security teams must govern not just model access, but the tools, memory and retrieval paths that let AI systems act inside production workflows.

Context

AI security problems divide into control over the model itself and control over how the model is used after deployment. The first covers poisoned memory, tampered checkpoints and contaminated training data. The second covers tool access, retrieval leakage and multi-agent behaviour that can turn a safe model into an unsafe system.

That distinction matters because many governance models still treat model safety as a boundary problem when the real exposure emerges inside application flows. Once an AI system can call tools, remember prior context and interact with other agents, the security question becomes identity, privilege and runtime control, not just content filtering.

Key questions

Q: What breaks when AI security is treated only as model security?

A: Model-only security misses the part of the system that actually touches tools, data, and workflows in production. A secure model can still produce unsafe outcomes if the surrounding agent, connectors, or permissions are not governed. Practitioners need controls that follow the operational identity, not just the model artefact.

Q: Why do agentic AI systems increase risk for API security and governance?

A: Agentic AI increases risk because it can interpret context, make decisions, and execute multi-step actions without waiting for direct human approval. That autonomy is useful, but it also expands the blast radius of a mistake, bad prompt, or weak permission boundary. If an agent can reach sensitive APIs or data, governance must assume both intended and unintended actions will occur.

Q: What signals show that AI runtime controls are failing?

A: Warning signs include unexplained tool usage, access to data outside the expected workflow, repeated policy overrides, and behavioural drift across sessions. If teams cannot trace why an action happened, who authorised it, and what context the system used, runtime controls are too weak to trust.

Q: Should organisations prioritise data hygiene or runtime testing for AI security?

A: They need both, but the priority depends on the exposure path. Data hygiene and checkpoint governance reduce model corruption, while runtime testing and access control reduce misuse after deployment. If the system is agentic, runtime testing usually deserves faster attention because that is where real-world compromise is most likely to emerge.

Technical breakdown

Model-layer poisoning and checkpoint tampering

Model-layer attacks target the artefacts that define model behaviour before deployment. Memory poisoning changes how the system responds over time, while backdoored fine-tuned checkpoints and unsafe data merges can plant malicious behaviour inside otherwise legitimate models. The practical security issue is not just incorrect outputs, but hidden behaviour that survives testing until a specific trigger appears. Governance here depends on dataset hygiene, version control, provenance checks and safe fine-tuning pipelines. Where teams use custom fine-tuned or foundation models, the attack surface includes both the training corpus and the model weights themselves.

Practical implication: Treat model training inputs and checkpoints as governed assets with provenance, review and rollback controls.

Runtime abuse through tools, memory and retrieval

Once an AI system is deployed, attackers often stop trying to corrupt the model and instead exploit the application layer around it. Tool-calling logic may grant unintended access, multi-turn memory can revive forbidden context, and retrieval layers can leak internal data that the model should never expose. In agentic systems, those failures compound because agents can cooperate, chain actions and operate across unpredictable workflows. This is where the question becomes one of non-human identity, delegated authority and runtime privilege scope. The model may be safe in isolation, but the surrounding system can still turn it into a data-exfiltration path or an unauthorised actor. See the Agentic AI Security Guide.

Practical implication: Constrain tool permissions, retrieval scope and agent authority as runtime controls, not as prompt-only safeguards.

Why static filters fail against adaptive AI attacks

Static controls work best when the threat is predictable, but AI systems create ambiguity, multi-step reasoning and emergent behaviour. Prompt injections, double-meaning sentences, false moral dilemmas and adversarial agent cooperation can all bypass a defence that only recognises known bad strings. The technical problem is that the attack path unfolds over several turns and sometimes across multiple components, so a single-pass policy check is insufficient. Effective testing therefore has to observe the system in motion, with repeated probing and state-aware attack sequences. That shifts AI defence from content moderation to behavioural validation.

Practical implication: Test AI systems with iterative adversarial paths, not one-shot prompt filters.

Threat narrative

Attacker objective: The attacker wants to convert an AI system's reasoning and tool access into data leakage, policy bypass or operationally harmful action.

1. Entry occurs when an attacker uses prompt injection, ambiguous instructions or manipulated retrieval content to influence the AI system's behaviour.
2. Escalation follows when the system grants unintended tool access, revives restricted context through memory, or coordinates with other agents in ways designers did not intend.
3. Impact occurs when the chain reaches data exposure, compliance failure or unsafe automated action inside the production application.

Breaches seen in the wild

• Langflow Flodrix botnet 2025: Attackers used Langflow's unauthenticated code endpoint to dump AI servers' environment variables and enlist them in the Flodrix botnet.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI security has split into model integrity and runtime governance, and treating those as the same problem is now a control failure. A model can be clean at training time and still become unsafe once tool calls, memory and retrieval are introduced. The article's two-layer framing is useful because it separates artefact protection from operational misuse, which is where agentic systems most often fail. Practitioners should map controls to the layer where the exposure actually occurs.

Non-human identity is now part of AI security, not a side issue. When an AI system can call tools, access retrieval stores and cooperate with other agents, it is operating with delegated authority that needs identity-bound governance. That makes the gap between identity, privilege and behaviour materially important, especially for teams using agentic AI security controls. Practitioners should stop treating the agent as a prompt wrapper and start governing it as an active runtime actor.

Model corruption and application abuse require different trust assumptions, so a single AI security control stack will always be incomplete. Dataset hygiene, version control and safe fine-tuning help with upstream integrity, but they do not stop a deployed agent from leaking data or misusing tools. Likewise, prompt filtering does little against poisoned training data or backdoored checkpoints. Practitioners should align assurance methods to the attack surface rather than to the technology label.

Static policy enforcement is losing to adaptive AI attacks because the threat unfolds over state, sequence and context. Prompt injection, multi-turn memory exploitation and adversarial agent cooperation are dynamic behaviours, not fixed signatures. That means detection and validation have to observe system behaviour over time, including how context changes across turns. Practitioners should move from one-pass review to continuous red-teaming and regression testing.

AI governance debt is accumulating wherever organisations deploy agents faster than they define their authority boundaries. The article's emphasis on business impact, compliance gaps and prioritised remediation reflects a broader market shift: security teams now need evidence that AI behaviours are bounded, testable and attributable. Practitioners should make AI identity, tool scope and runtime logging part of governance evidence, not an afterthought.

From our research library:

• Only 23% of IT leaders were very confident in their organisation's ability to manage security and governance for GenAI deployments, according to a 2025 Gartner survey of 360 IT leaders.
• Read next: Agentic AI Security Guide

What this signals

Agentic AI governance debt: organisations are now accumulating risk wherever they deploy systems that can remember, decide and act before they define who may authorise those actions. The practical question is no longer whether the model is safe in isolation, but whether the surrounding workflow can prove where authority begins and ends.

The next programme shift is from prompt-centric assurance to behaviour-centric assurance. That means runtime logging, tool inventory, memory scoping and regression testing become part of governance evidence, especially where agents handle sensitive data or compliance-relevant decisions.

For practitioners

• Define separate controls for model and runtime risk Document which controls apply to training-time integrity and which apply to deployed application behaviour. This prevents teams from assuming that one set of safeguards covers memory poisoning, checkpoint tampering, tool misuse and retrieval leakage equally.
• Inventory every tool and retrieval path an agent can reach List the exact tools, connectors, memory stores and retrieval sources available to each agentic workflow, then classify each one by data sensitivity and permitted action scope.
• Restrict delegated access to the minimum runtime scope Limit each agent to the smallest set of actions, data sources and session permissions required for the task, and remove broad standing access from production flows.
• Test agents with adaptive attack chains Use repeated prompt injection, multi-turn memory abuse and adversarial cooperation scenarios in regression testing so controls are exercised under realistic runtime conditions.
• Tie remediation to compliance and operational impact Map AI security findings to concrete business consequences, including regulatory exposure, data handling failures and workload-specific control gaps, so prioritisation reflects actual risk.

Key takeaways

• AI security failures increasingly arise after deployment, when tools, memory and retrieval turn a competent model into an exposed system.
• The article draws a clear line between upstream model integrity controls and downstream runtime controls, and both are required.
• Practitioners should govern agent authority, test behaviour over time and map AI findings to business and compliance impact.

Related NHI Mgmt Group resources: Agentic AI Security Guide · Agentic AI Compliance Guide · AI Risk Assessment Guide · AI Agent Identity Security Buyer's Guide

Key terms

• Agentic AI: Autonomous AI systems capable of planning, deciding, and taking actions, including calling APIs, writing code, and orchestrating other agents, with minimal human oversight. Agentic AI introduces new NHI risks as agents must authenticate to external services.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Memory Poisoning (ASI06): An attack where malicious content is injected into an AI agent's memory or context, causing it to alter its behaviour in subsequent tasks, potentially exfiltrating secrets, escalating privileges, or acting against its intended purpose.
• Post-retrieval leakage: Exposure that happens after a secret is fetched from a vault or secret store. The secret may appear in logs, environment variables, shells, or debug output, which means storage security alone does not protect the full credential lifecycle.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security and agentic AI identity for practitioners who need to connect access control to runtime behaviour. It helps security and identity teams translate emerging AI risk into governance patterns they can operationalise.