LLM security is now a governance problem, not just a model problem

At a glance

What this is: Anthropic's August 2025 Threat Intelligence Report, as discussed by Noma Security, shows that LLM misuse is already spanning extortion, fraud and malware workflows.

Why it matters: It matters because security teams now need governance over model inventory, adversarial testing and runtime enforcement, not just model-centric safety reviews.

Context

LLM security is no longer just about model behaviour in isolation. Once adversaries can use a model to gather credentials, draft tailored lures, or sustain fraudulent activity, the control problem shifts to governance over access, usage, and runtime enforcement across the AI system.

The article sits at the intersection of AI governance and cybersecurity operations. For identity and access teams, the practical issue is not only what the model can generate, but who or what can invoke it, what data it can reach, and how those interactions are monitored and constrained.

Key questions

Q: How should security teams govern LLMs that can trigger tools or workflows?

A: Treat the LLM as an untrusted decision component, not an authorizer. Give it the minimum tool scope required, enforce policy outside the model, and require logging for every action it can influence. If the model can initiate work, then privilege, approval, and revocation controls must sit around it, not inside it.

Q: Why do LLMs increase the risk of fraud and extortion operations?

A: LLMs reduce the skill and time required to draft lures, automate reconnaissance, impersonate competence and tailor extortion content. That lowers the barrier for criminals and increases throughput, which is why the risk is not just better text generation. The security problem is scalable workflow enablement.

Q: What are the signs that an LLM’s safety controls are failing in practice?

A: Common warning signs include toxic or biased responses, inconsistent refusal behavior, over-acceptance of harmful prompts, and weak handling of demographic-sensitive content. If the model appears helpful but cannot reliably distinguish safe from unsafe output, the control layer is too loose. Teams should look for these symptoms across evaluation sets, not only in isolated demonstrations or benchmark scores.

Q: How do identity and privilege controls apply to agentic AI workflows?

A: They should define what the agent may access, what it may change, and which actions require stronger telemetry or approval. The useful question is not whether the user was authenticated, but whether the delegated authority given to the agent was proportional to the task and visible at the point of execution.

Technical breakdown

Why LLM misuse becomes a governance issue

Large language models become a governance problem when they are embedded into attack workflows, not just when they produce unsafe outputs. In this article, the model is used to support reconnaissance, credential harvesting, extortion drafting, fraud, and malware enablement. That changes the control objective from content moderation to oversight of model inventory, access paths, data exposure, and permitted use cases. Once an LLM sits inside operational workflows, security teams need to know which systems can call it, what it can see, and how abuse is detected before the model becomes an attack amplifier.

Practical implication: treat LLM access, data exposure and usage policy as governance controls, not only model-safety issues.

How red teaming and adversarial testing fit LLM security

Red teaming in LLM security means actively probing the model, surrounding orchestration, and policy layer for prompt injection, jailbreaks, misuse paths, and unsafe task completion. The value is not abstract assurance. It is finding where the model can be steered into helping with criminal workflows, or where guardrails fail under realistic adversarial pressure. For agentic or tool-using systems, the test surface expands to delegated actions, external calls, and chained prompts. That makes adversarial testing a control for both model behaviour and the identity and privilege of the systems around it.

Practical implication: test both the model and the surrounding orchestration for misuse paths before attackers do.

Runtime LLM security and the model in motion

Runtime LLM security focuses on what the model does during live use, not in a lab. It involves monitoring prompts, outputs, tool calls, and behavioural patterns so harmful activity can be blocked or constrained while the session is active. That is materially different from pre-deployment review because misuse often emerges only when the model is connected to data, tools, and real users. In practice, runtime controls sit closer to detection and prevention than to policy writing. They matter most where LLMs can generate action, not just text, because the risk becomes operational rather than theoretical.

Practical implication: instrument live model sessions with policy enforcement, telemetry and blocking controls.

Threat narrative

Attacker objective: The objective is to scale criminal operations by using LLMs to lower skill barriers, increase throughput and improve the credibility of fraud, extortion and malware workflows.

1. Entry occurs when attackers use LLMs to support reconnaissance, scam creation, or automated workflow assembly inside legitimate-looking activity.
2. Credential access and trust exploitation follow when the model helps harvest credentials, simulate technical competence, or generate victim-specific lures and extortion content.
3. Escalation and impact occur when those assisted workflows scale across multiple organisations, enable malware and ransomware packaging, or sustain fraudulent employment and monetisation schemes.

Breaches seen in the wild

• AI LLM hijack breach: attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.
• LiteLLM PyPI package breach: LiteLLM PyPI supply chain attack, credentials stolen from users.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM governance has shifted from model quality to operational control. Once an LLM is used inside fraud, extortion or malware workflows, the core risk is no longer only unsafe content generation. The real issue is who can invoke the model, what data it can reach, and whether its outputs can trigger harmful actions. Practitioners should treat model inventory and access governance as first-order security controls.

AI red teaming now belongs in the control stack, not the assurance appendix. The report's cases show the same principle defenders already know from offensive security testing: if adversaries can shape the system into doing work for them, policy claims are not enough. Structured adversarial testing is how organisations expose prompt injection paths, tool abuse and unsafe delegation before those paths become operational incidents. That makes red teaming a practical control for both AI governance and identity governance where systems act on behalf of users or services.

Runtime enforcement is the new boundary for LLM risk. Static review cannot stop a live model from drifting into harmful execution when it has access to tools, data or downstream automation. Runtime misuse detection: the article shows that the decisive failure mode is allowing harmful use to unfold inside a live session without monitoring or intervention. Security teams should frame runtime telemetry and blocking as the equivalent of model-side detection and prevention.

Agentic AI will inherit the same governance gaps unless identity is explicit. The article implies a broader pattern: whenever AI systems can execute tasks, not just generate text, identity, privilege and authorization become inseparable from AI security. That is where NHI and agentic AI governance intersect most directly. If a model or agent can call tools, reach data or carry out workflows, the organisation must govern that runtime identity with the same seriousness it applies to privileged human access. Practitioners should align AI governance with explicit identity and privilege controls.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Organisations that rely heavily on static credentials reported a 20-percentage-point increase in security incidents compared with those with low reliance, according to the 2026 Infrastructure Identity Survey.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

LLM governance will increasingly be measured by whether organisations can constrain action, not merely content. As models move deeper into workflows, teams should expect pressure to connect AI policy to identity, privilege and runtime enforcement. The governance gap is not theoretical: if a model can reach tools or data, the control surface has already expanded beyond classic model review.

Agentic AI identity is now part of the LLM security conversation. The moment a system can act, the question changes from what the model said to what it was allowed to do. That means security teams should prepare to govern AI connectors, service accounts and delegated privileges with the same discipline used for other non-human identities.

For practitioners

• Map every model and agent to an inventory Track which LLMs are in production, what data they can access, which tools they can invoke, and who is authorised to trigger them.
• Test adversarial workflows before deployment Run structured red-team exercises against prompt injection, harmful tool use, delegated actions and criminal workflow simulation.
• Instrument live sessions for misuse detection Monitor prompts, outputs and tool calls in production so risky behaviour can be flagged or blocked while a session is active.
• Treat AI-access pathways as privileged Apply least-privilege principles to model connectors, agent permissions and downstream service accounts that can be invoked by AI systems.

Key takeaways

• LLM security now spans governance, red teaming and runtime controls because attackers are using models inside real criminal workflows.
• The report's cases show that AI lowers the skill threshold for extortion, fraud and malware enablement while increasing operational scale.
• Security teams need explicit inventory, authorization and live monitoring for models and agents, or AI becomes an attack accelerator.

Key terms

• LLM Governance Debt: LLM governance debt is the gap between rapid deployment of model-enabled workflows and the slower work of defining ownership, access control, auditability, and response procedures. It accumulates when teams scale use cases before establishing the controls needed to manage model behaviour safely.
• Runtime LLM Security: Runtime LLM security is the set of controls that observe and enforce policy while an AI model is actively in use. It focuses on prompts, outputs, tool calls, and access to external data so defenders can block misuse during live operation, not only during pre-deployment testing.
• AI bill of materials: An inventory of the components used by an AI development environment, including models, plugins, hooks, skills, and agent integrations. It gives security teams visibility into what is operating in the stack so they can assess trust, dependency risk, and governance coverage.
• Adversarial AI: Adversarial AI refers to AI used by attackers to scale reconnaissance, impersonation, or abuse in ways that overwhelm normal manual review. For defenders, the issue is not just malicious model use, but the way machine-speed behaviour changes the timing, volume, and accuracy demands on identity and fraud controls.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, helps practitioners connect identity governance, machine identity security and agentic AI identity to operational controls. It is designed for teams that need to govern privileged non-human access across modern environments.