A fake function definition is untrusted schema text supplied in a prompt that imitates a real tool description. Attackers use it to confuse the model into treating user content like an approved function, which can expose internal instructions, leak hidden context, or trigger unsafe tool-related behavior.
Expanded Definition
A fake function definition is a prompt injection technique that places fabricated tool or schema text into untrusted input so a model mistakes attacker-supplied content for an approved interface. The goal is to blur the boundary between user content and system-controlled function metadata, especially in environments where tool calling, structured outputs, or agent orchestration are in use.
This is distinct from ordinary prompt manipulation because the attacker is not only asking the model to reveal information or change behavior, but is also impersonating the structure of the runtime itself. In practice, the risk increases when applications concatenate retrieved text, user messages, and developer instructions into a single context without clear separation. Security teams should treat any schema-like block, function signature, or tool description inside untrusted content as hostile unless it is explicitly validated and injected by the application. The most common misapplication is assuming that any text formatted like a function definition is safe to execute or reference when it appears inside a user-controlled prompt.
Examples and Use Cases
Implementing defenses against fake function definitions often introduces strict parsing and content separation constraints, requiring organisations to balance agent flexibility against stronger validation and review.
- A support chatbot ingests a pasted document that includes a fake tool specification and then begins following attacker-authored instructions as if they were application-defined actions.
- An agentic workflow reads retrieved web text containing a counterfeit JSON schema, causing the model to treat attacker-controlled fields as legitimate tool arguments.
- A coding assistant sees prompt content that imitates a plugin manifest and responds as though the user had approved access to hidden function capabilities.
- A retrieval-augmented generation system forwards raw source material into the model context without isolating metadata, allowing untrusted schema text to compete with trusted instructions.
- A security review process flags structured prompt content that resembles tool syntax, then strips or normalises it before the model receives the input.
For teams building agent workflows, guidance from the NIST Cybersecurity Framework 2.0 is useful because it reinforces governance, control visibility, and resilience around system inputs that can influence decision-making.
Why It Matters for Security Teams
Fake function definitions matter because they target the trust boundary between natural language and executable action. Once a model confuses untrusted text with approved tool metadata, the impact can include hidden instruction leakage, improper tool invocation, or escalation into actions the operator never intended. That risk is especially sharp in agentic AI systems where function calling is tied to real-world operations such as ticket creation, data retrieval, approvals, or account changes.
For security teams, the practical issue is not just prompt quality but control of provenance. Tool definitions should come from code, policy, or signed configuration, not from user-supplied content or retrieved documents. This is where AI security and identity governance intersect: if an agent can impersonate a trusted function, it can also impersonate a trusted request path. Organisations typically encounter the operational impact only after an agent has already disclosed protected context or executed an unsafe action, at which point fake function definition handling becomes operationally unavoidable to address.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and CSA MAESTRO address the attack and risk surface, while NIST AI RMF, NIST CSF 2.0 and NIST IR 8596 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | Covers tool misuse and prompt injection patterns in agentic AI systems. | |
| NIST AI RMF | Defines governance practices for managing AI risks, including prompt abuse. | |
| NIST CSF 2.0 | GV.OC-01 | Supports organisational understanding of AI system context and trust boundaries. |
| NIST IR 8596 | Addresses cyber risks in AI systems, including adversarial input handling. | |
| CSA MAESTRO | Provides security guidance for agentic workflows and tool-using AI systems. |
Separate untrusted text from tool specs and validate every function call against policy.
Related resources from NHI Mgmt Group
- What is the difference between function calling and MCP for enterprise security?
- When does MCP make more sense than function calling?
- What is the difference between application RBAC and function-level permissions for MCP?
- Why do unsalted password hashes remain risky even when the hash function is strong?
