What breaks when AI agents accept user-supplied function schemas or parameter names?

Why This Matters for Security Teams

Function schemas and parameter names are not harmless interface details when an AI agent can read and act on them. They shape what the agent believes it is allowed to do, what it can reveal about its own toolset, and how it routes user input into downstream actions. That turns a naming layer into a security boundary, which is why OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework both matter here: the issue is not just model accuracy, but control integrity and misuse resistance.

Practitioners often underestimate how quickly a schema becomes an attack surface. If user input can rename arguments, alter descriptions, or inject seemingly valid tool definitions, the agent may treat attacker-chosen content as operational context. That can expose hidden instructions, weaken consent checks, and create a path from prompt manipulation to tool abuse. In practice, many security teams encounter this only after a benign-looking field rename has already been used to steer a live agent into unauthorized behaviour, rather than through intentional testing.

How It Works in Practice

The failure mode usually appears when an application lets users influence the structure of a tool call instead of only the values passed into a fixed schema. At that point, the model is no longer choosing among trusted parameters. It is evaluating attacker-controlled names, descriptions, or object shapes and may map them to internal tooling, logging, or policy logic. That makes the boundary between data and instruction much weaker than developers expect.

Security teams should treat the schema as part of the trust perimeter. A practical implementation pattern is to keep function definitions server-side, pin parameter names and types, and validate every tool invocation against an allowlisted contract before execution. When agent workflows need dynamic fields, the safest approach is to map user input into pre-declared slots rather than accepting arbitrary schema text. It also helps to separate natural-language prompts from tool metadata so the model cannot confuse one for the other.

• Freeze function names, argument names, and descriptions in code or a trusted registry.
• Reject any user-supplied schema elements that attempt to redefine tools or add new ones.
• Normalize and validate parameters before the model sees them, not after execution starts.
• Log tool-selection decisions so unexpected schema influence can be investigated quickly.

Where the question becomes operationally serious is in agents connected to tickets, cloud APIs, identity systems, or data retrieval layers. A manipulated parameter name can shift the agent from answering a question to performing an action, and that is where consent, authorization, and auditability can all fail at once. Current guidance suggests treating schema acceptance as a high-risk design choice, not a convenience feature, because the same weakness can support disclosure, spoofed intent, and unauthorized side effects. These controls tend to break down when developers generate schemas on the fly for multi-tenant agents because policy checks are often applied too late in the request path.

Common Variations and Edge Cases

Tighter schema control often increases development overhead, requiring organisations to balance flexibility against predictability. That tradeoff matters most in systems that need extensible plugins, customer-defined automations, or rapid agent prototyping. Best practice is evolving here, and there is no universal standard for allowing partial schema dynamism without creating a manipulation channel.

One edge case is benign user personalization, such as letting a user label a field or rename a display element. That can be acceptable if the label never reaches the model as executable tool metadata. Another is retrieval-augmented workflows where the agent sees external JSON or OpenAPI-like content; those inputs should be treated as untrusted data unless a trusted parser and schema governor sit in front of them. The same caution applies when model outputs are fed into another agent, because one compromised schema can cascade through a chain of delegated actions.

For teams building agentic systems, the broader lesson is simple: if a user can shape the tool contract, they may also shape the agent’s sense of authority. NHI Management Group recommends reviewing not just prompts and permissions, but every place where names, descriptions, or parameter structures can be influenced by untrusted input.

Related resources from NHI Mgmt Group

• What breaks when AI SOC agents cannot map local SIEM schemas correctly?
• What breaks when AI agents inherit user OAuth sessions too broadly?
• What breaks when AI agents are detected only through package names or file checks?
• How should security teams use AI agents for user access reviews?