LLM function-calling security needs least privilege and boundary separation

At a glance

What this is: This article explains the security risks created when LLMs are given external tool-calling abilities, with a focus on SSRF, prompt injection, excessive delegation, and credential leakage.

Why it matters: IAM, NHI, and platform security teams need to treat tool credentials, context boundaries, and delegated permissions as governance problems because LLM integrations can turn ordinary access into unsafe autonomous action.

By the numbers:

• Only 5.7% of organisations have full visibility into their service accounts.
• NHIs outnumber human identities by 25x to 50x in modern enterprises.
• 96% of organisations store secrets outside of secrets managers in vulnerable locations including code, config files, and CI/CD tools.

👉 Read Flatt Security's analysis of LLM function-calling risks and mitigations

Context

LLM function-calling becomes a security issue the moment the model can reach beyond text generation and invoke tools that read data, trigger API actions, or pass information into other systems. The primary problem is not the model alone, but the trust boundary around the tool, the credentials behind it, and the data that is allowed into the context window.

For identity programmes, this is an NHI problem as much as an application-security problem. Tool tokens, API keys, and service permissions can be over-scoped, exposed through prompts, or misused when external content is treated as trusted input. The article is typical of the current generation of LLM integration guidance, which is useful precisely because these designs are becoming normal rather than exceptional.

The security question is therefore whether the surrounding architecture can prevent the model from turning access into action without crossing policy boundaries. That is where least privilege, separation of credentials, and context isolation become identity controls rather than implementation details.

Key questions

Q: What breaks when LLMs can call external tools without strict boundaries?

A: The main failure is that the model can convert untrusted text into real action. Without strict boundaries, prompt injection, SSRF, and over-scoped credentials can push the system into unintended requests, data exposure, or destructive writes. The issue is not the LLM alone, but the delegated access path around it.

Q: Why do tool credentials increase risk in LLM applications?

A: Tool credentials turn model suggestions into authorised operations. If those credentials are broad, the LLM can do far more than the user intended when it is misled by ambiguous prompts or malicious content. Narrow, task-specific scopes reduce the blast radius of both user error and injection attacks.

Q: How can security teams tell whether an LLM tool integration is too permissive?

A: A tool integration is too permissive when a mistake or injected instruction could reach unrelated repositories, hidden data, or admin-level actions. If the tool identity can write, delete, or fetch beyond the immediate task, the permissions are already wider than the business need.

Q: Should organisations separate browsing, retrieval, and action tools in LLM systems?

A: Yes. Retrieval tools, browsing tools, and action tools should have different trust levels because they do not carry the same risk. Keeping them separate limits how far untrusted content can travel and prevents a read-only task from becoming an unintended write operation through the same identity.

Technical breakdown

Tool calling turns LLM output into real system actions

Tool calling, also called function calling, lets an LLM emit structured output that an application translates into API requests or other operations. The model does not execute the action itself, but it does influence what action is chosen, what arguments are passed, and how outside data returns into the next prompt cycle. That makes the tool layer a security boundary, not a convenience layer. Once an LLM can invoke URLs, issue comments, or create records, the risk surface includes request forgery, untrusted input handling, and policy bypass through indirect instructions.

Practical implication: Treat every registered tool as a privileged interface and review its permission scope as part of identity design.

Prompt injection can convert external content into malicious instructions

Prompt injection appears when untrusted content contains instructions that the model may follow as if they were part of the task. In retrieval or URL-fetch patterns, the model can be induced to request internal resources, leak data, or change behaviour based on embedded text. The key issue is boundary confusion: the LLM processes content and instructions in the same conversational flow unless the application explicitly separates them. This is why the article stresses clear input and output boundaries, along with context discipline.

Practical implication: Mark untrusted content as untrusted in the application layer and keep instruction channels separate from retrieved text.

Least privilege must apply to tool credentials and context exposure

When a tool uses a PAT, service token, or other credential, the permissions behind that identity determine how far a mistaken or malicious action can go. Over-broad access means the model can create, modify, or delete far more than intended if it is misled by ambiguous prompts or hostile context. Separation of credentials is critical because putting secrets into the prompt path turns the context window into a leakage channel. The article’s strongest point is that policy enforcement must happen outside the LLM, in trusted software logic, where access scope can be constrained deterministically.

Practical implication: Issue task-specific credentials with narrow scopes and keep them out of the LLM context entirely.

Threat narrative

Attacker objective: The attacker wants to turn the LLM’s delegated tool access into unauthorized data exposure or unsafe system action.

1. Entry occurs when an attacker supplies a malicious URL, document, or prompt payload that the LLM processes as part of tool-enabled workflow.
2. Escalation happens when prompt injection or over-scoped tool credentials cause the application to retrieve internal data, issue unintended requests, or perform actions beyond the user’s intent.
3. Impact is achieved when the model leaks sensitive information, manipulates external systems, or executes destructive changes through the delegated tool path.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Function-calling security is really NHI governance in disguise: once an LLM can act through a tool identity, the question is no longer whether the model is fluent, but whether the delegated credential is bounded. The article shows that the security problem sits in the interface between model output and external action, where tokens, APIs, and context boundaries determine what the system can do. That means identity governance, not model quality, becomes the durable control plane for AI-enabled workflows.

Context window separation is a governance control, not a prompt-tuning trick: the article’s strongest architectural point is that information entering the context window should be treated as potentially exposed and potentially influential. That is a broken assumption for many current LLM designs, because the same channel carries user intent, retrieved content, and operational instructions. Practitioners should read that as a boundary-design problem, not a language-model problem.

Excessive delegation is the failure mode to name, not just a risk to monitor: when a tool identity can create issues, post comments, or fetch arbitrary URLs, the model inherits privileges that may exceed the human task. That is a form of privilege drift at the machine layer, and it aligns closely with OWASP-NHI and zero-trust assumptions about scoped access. The practical conclusion is that task specificity must be enforced before the model ever sees the credential.

Least privilege only works if the tool is narrow enough to be meaningfully privileged: the article correctly warns that generic browsing or broad action tools create unpredictable behaviour. That is where NHI governance and application design intersect, because a broad tool identity cannot be safely recertified or reasoned about in the same way as a tightly bounded service account. Security teams should treat overly broad tool design as a governance defect, not merely an implementation choice.

From our research:

• Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs.
• From our research: 91.6% of secrets remain valid five days after the targeted organisation is notified, according to the Ultimate Guide to NHIs.
• From our research: See also The 52 NHI breaches Report for the recurring breach patterns that follow secret exposure and poor lifecycle control.

What this signals

Context window separation: the model may be the executor, but the security boundary still lives in the surrounding identity architecture. As LLM applications move from read-only retrieval to write-capable tool use, teams need to classify every credential, token, and external API as an NHI with explicit scope and review.

The practical signal for IAM and platform teams is that AI integrations will increasingly expose gaps already familiar from service-account governance: poor visibility, over-privilege, and weak offboarding. Where the tool can act, the identity programme must decide what it can see, what it can touch, and what it can never infer.

That is why zero-trust thinking belongs at the tool layer as much as the network layer. If the application cannot prove that a prompt, document, or URL is safe to influence action, then the default should be no delegated execution at all.

For practitioners

• Separate tool credentials from the LLM context Keep API keys, PATs, and service tokens in trusted application logic rather than passing them through prompts or retrieval flows. That reduces both leakage risk and the chance that the model can disclose or misuse secrets during tool execution.
• Constrain each tool to one narrow job Prefer task-specific tools such as a pull-request creator over generic browser or admin interfaces. Narrow tools make it easier to reason about allowable actions and reduce the damage if prompt injection reaches the workflow.
• Treat retrieved content as untrusted input Isolate fetched web pages, repository text, and document content from instruction channels, and do not let them silently influence tool selection or credential use. Clear separation between data and instructions is essential.
• Review delegated scopes before enabling automation Check whether the LLM’s assigned permissions match the smallest possible task scope, especially for write actions in Git systems or external APIs. If a human would not approve the action without context, the tool scope is too broad.

Key takeaways

• LLM function-calling creates an identity problem as much as an application problem because tool credentials, context, and action scope determine what the model can really do.
• The most dangerous failures are prompt injection, SSRF, and excessive delegation, because they turn untrusted content into authorised external action.
• Least privilege, credential separation, and strict context boundaries are the controls that keep delegated AI workflows inside a governable risk envelope.

Key terms

• Tool calling: Tool calling is the pattern where a model selects and invokes an external function during runtime. In agent systems this turns text generation into action execution, so the access decision must be constrained, logged, and governed like any other privileged interaction.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Context Window Instruction Data Separation: Context window instruction data separation is the challenge of keeping trusted instructions distinct from untrusted retrieved content inside a single LLM input stream. Because models often receive both in the same channel, defenders must add structure, parsing, or policy controls to prevent data from being misread as commands.
• Excessive Delegation: Excessive delegation occurs when an LLM or its tool identity is granted more authority than the task needs. The result is widened blast radius, because a single mistaken or malicious prompt can trigger actions, access paths, or repository changes that should never have been available to the workflow.

What's in the full article

Flatt Security's full blog post covers the operational detail this post intentionally leaves for the source:

• Concrete request-flow examples for URL fetching and Git hosting integrations, showing where trust boundaries are crossed.
• Specific countermeasure patterns for SSRF, prompt injection, and indirect prompt injection in tool-calling designs.
• Design guidance for separating credentials from prompts and keeping secrets out of the context window.
• Examples of fine-grained versus overly broad token scopes for Git-based workflows.

👉 Flatt Security's full post covers the attack paths, tool-flow examples, and boundary controls in more detail.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.