LLM security is now a governance problem, not just a model problem

At a glance

What this is: Anthropic’s August 2025 Threat Intelligence Report shows how adversaries are already using LLMs for extortion, fraud, and other criminal workflows, with governance and runtime controls emerging as the practical response.

Why it matters: For IAM, NHI, and security teams, this matters because AI systems are becoming access-bearing systems that need visibility, policy, and monitoring like any other privileged workload.

👉 Read Noma Security's analysis of Anthropic's August 2025 threat intelligence report

Context

LLM security is moving from a model-quality issue to a governance and access-control problem. Once an AI system can browse, generate, call tools, or help coordinate abuse, the question is no longer only what the model says, but what it can do and what data it can reach. That creates an identity and privilege problem as much as an AI one.

Anthropic’s report frames a wider market reality: criminal use of LLMs is becoming operational, not theoretical. For security leaders, the implication is that inventory, policy enforcement, and runtime oversight must be applied to AI systems with the same seriousness as other high-risk digital assets.

Key questions

Q: How should security teams govern AI models that can call tools and access data?

A: Security teams should govern AI models as non-human identities with named owners, limited scope, short-lived credentials, and continuous authorization. The critical shift is to treat every tool call, data read, and update path as a privileged action that can be logged, revalidated, and revoked. Without that discipline, model risk becomes identity risk.

Q: Why do conversational AI systems create new identity and access risks?

A: Because they can combine data retrieval, decision-making, and execution in a single interaction. That collapses the gap between information access and business action, which traditional IAM and security tools were not built to manage. The result is higher exposure when the system can modify records or disclose sensitive guest data.

Q: What do enterprises get wrong about AI red teaming maturity?

A: Many teams stop at attack simulation and assume the test itself is the control. In practice, maturity depends on whether findings feed monitoring, policy enforcement, and audit-ready reporting in production. A strong programme reduces risk after the test, not just during the assessment window.

Q: Who is accountable when an AI system makes a harmful decision?

A: Accountability should follow the identity chain that authorized, configured, or triggered the action, including the human owner, the platform team, and any delegated agent or tool account. If the organisation cannot name that chain, the governance model is too weak for regulated AI use.

Technical breakdown

AI-driven extortion workflows and delegated abuse

LLM-enabled extortion works by compressing tasks that once required specialist skill. An attacker can use an LLM to automate reconnaissance, draft targeted lures or ransom notes, and tailor follow-on actions across many victims at speed. The model is not the attacker’s objective by itself. It is an execution layer that helps structure the attack chain and reduce operator effort. That makes misuse harder to spot with controls that only look for human-like patterns or obvious malware. In governance terms, the issue is not just content safety. It is whether a model can be used to amplify reconnaissance, credential abuse, and extortion at operational scale.

Practical implication: classify LLMs that can touch internal data or tools as governed assets with explicit use boundaries and logging.

AI governance and the case for an AIBOM

An AI Bill of Materials, or AIBOM, is the inventory of the components, data sources, prompts, connectors, and dependencies that shape an AI system. It matters because LLM risk is rarely confined to the base model. Exposure can come from embedded secrets, connected APIs, retrievers, plugins, or unmanaged copies of the same model embedded in different workflows. Without that inventory, teams cannot answer basic questions about where the model runs, what it can access, or who can change its behavior. That is a control failure, not just a documentation gap.

Practical implication: maintain an AIBOM for each production AI use case and tie it to access reviews, change control, and incident response.

Runtime LLM security and policy enforcement

Runtime LLM security applies monitoring and enforcement while the model is in use. That includes observing prompts, outputs, tool calls, and abnormal usage patterns, then blocking actions that violate policy or signal misuse. This is distinct from pre-deployment testing because threat behavior changes once the system is live and connected to real data, real users, and real tools. For defenders, the main architectural challenge is deciding which actions can be logged, which can be slowed, and which must be denied immediately. The control objective is to stop harmful activity mid-flight, not simply to detect it after the fact.

Practical implication: define runtime guardrails for tool use, data access, and escalation paths before AI systems are allowed into production.

Threat narrative

Attacker objective: The attacker aims to increase the speed, scale, and success rate of criminal operations while lowering the skill required to execute them.

1. Entry occurs when adversaries use LLMs to accelerate reconnaissance, craft social engineering, or operationalise fraud and extortion.
2. Escalation follows when the model is used to automate credential harvesting, employment fraud, or repeated attack steps at scale.
3. Impact lands as targeted extortion, financial crime, or broader abuse that is faster, cheaper, and easier to run than traditional human-led operations.

NHI Mgmt Group analysis

LLM security is now an access-governance problem, not just a content-safety problem. Once models can retrieve data, call tools, or support decisions, they become systems with effective privileges. That means the governing question is what an AI system can reach, change, or trigger, not just whether its outputs are safe. Practitioners should treat model access as a governed entitlement boundary.

AI governance debt is accumulating faster than most organisations can inventory it. The article’s emphasis on complete visibility mirrors the real operational gap: teams often cannot tell where models run, what connectors they use, or which datasets they can touch. That gap becomes a compliance and incident-response problem the moment misuse occurs. Practitioners should close inventory and ownership gaps before adding more AI use cases.

Runtime LLM security is the control layer that turns policy into enforcement. Red teaming matters, but it is only a pre-production check. The article correctly points to the need for monitoring and blocking behaviour during live use, which is where abuse actually happens. Practitioners should move from static approval models to live control points that can interrupt misuse mid-session.

Adversaries are turning LLMs into force multipliers for fraud and extortion. The central shift is economic: criminals no longer need deep expertise to generate plausible tradecraft, scale outreach, or tailor attacks. That lowers the barrier to entry and widens the attacker pool. Practitioners should re-evaluate threat models that still assume human-speed adversaries.

Identity controls have to extend into AI systems that act on behalf of users and services. As soon as an LLM or agent can invoke tools, fetch records, or trigger workflows, it becomes part of the identity and privilege estate. That is where IAM, PAM, and NHI governance intersect with AI security. Practitioners should align AI oversight with existing entitlement and secrets governance rather than treating it as a separate programme.

What this signals

LLM governance is starting to resemble non-human identity governance. Once an AI system can act, retrieve, and delegate, it needs the same visibility and access discipline that security teams apply to service accounts and other privileged workloads. The practical shift is to put model ownership, permissions, and secrets under the same operational control plane used for other high-risk identities.

AI governance debt will surface first as access sprawl. Many organisations are adding connectors, plugins, and credentials faster than they can map them. The result is a control gap that sits between AI experimentation and production risk management, and it will become harder to unwind later. Teams should prepare for tighter review of AI-connected entitlements and clearer ownership mapping.

Runtime controls will matter more than policy statements. NIST AI 600-1 and the OWASP Agentic AI Top 10 both point toward operational controls that can observe and constrain live behaviour. For practitioners, that means moving from AI approval lists to enforceable guardrails on tool use, retrieval, and external action.

For practitioners

• Inventory AI systems as governed assets Create a complete register of production and shadow AI use cases, including data sources, connectors, secrets, and tool permissions. Tie each system to an owner, a purpose, and an approved access scope. This is the only way to answer what the model can actually do.
• Build an AIBOM for every production use case Document the model, training and retrieval dependencies, prompt templates, API connections, and external services behind each AI deployment. Update the AIBOM through change control so incident responders can trace exposure quickly.
• Enforce runtime policy on tool use and data access Set explicit allow and deny rules for prompt handling, tool invocation, output filtering, and data retrieval. Where risk is high, add blocking controls instead of relying only on after-the-fact review.
• Stress-test AI workflows with adversarial scenarios Run red-team exercises against prompt injection, credential harvesting, fraud workflows, and tool misuse. Focus the testing on how the model behaves once it is connected to live data and downstream systems.
• Extend identity governance into AI-connected systems Review whether service accounts, API keys, and delegated credentials used by AI systems have standing privilege or unclear ownership. Apply least privilege and lifecycle controls to every AI-connected identity.

Key takeaways

• LLM misuse is now a live criminal workflow issue, not a theoretical model-risk discussion.
• The strongest defence combines inventory, governance, red teaming, and runtime enforcement across AI-connected access paths.
• Identity and privilege controls must extend into AI systems, or the model becomes another unmanaged access surface.

Key terms

• AIBOM: An AI Bill of Materials is a structured inventory of the components, data sources, prompts, connectors, and dependencies that shape an AI system. It helps security teams understand what the model can access, where risk enters the stack, and which changes require governance review.
• Runtime LLM Security: Runtime LLM security is the set of controls that observe and enforce policy while an AI model is actively in use. It focuses on prompts, outputs, tool calls, and access to external data so defenders can block misuse during live operation, not only during pre-deployment testing.
• AI Governance: AI governance is the set of controls used to discover, classify, approve, restrict, monitor, and revoke AI-enabled access. It connects identity, data, and policy so organisations can manage what AI can reach, what it can share, and when it should be stopped.
• Agentic AI: Autonomous AI systems capable of planning, deciding, and taking actions — including calling APIs, writing code, and orchestrating other agents — with minimal human oversight. Agentic AI introduces new NHI risks as agents must authenticate to external services.

What's in the full article

Noma Security's full article covers the operational detail this post intentionally leaves for the source:

• Case-by-case examples of how Claude was abused across extortion, fraud, and malware workflows.
• The specific detection and classification approaches the vendor used to identify suspicious usage patterns.
• The incident response actions the vendor took, including account bans and intelligence sharing.
• The article’s full discussion of AI governance and runtime security controls in practice.

👉 Noma Security's full post expands the case studies, governance model, and runtime security implications.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and the identity controls that underpin AI-connected systems. It is designed for practitioners who need a common governance language across identity, access, and emerging agentic workloads.