Structured Outputs vs Free Text for Agent Actions: What Schemas Stop and What They Miss

Teams that move agent actions from free text to schema-constrained structured output often believe they have solved injection, when they have only solved parsing. A JSON Schema guarantees that a tool call has the right shape: the right fields, types and allowed values. It says nothing about whether the values are what the user wanted, whether the caller may perform the action, or whether a prompt injection chose them. This guide compares free text and structured output as the interface between a model and the systems it acts on, sets out the threat model for each, shows the valid-but-malicious failure mode that schemas cannot catch, and describes the validation and authorisation layers that must sit behind them. It is for application security engineers, AI platform teams and architects building agents. It goes beyond the OWASP LLM Top 10 LLM05 Improper Output Handling Guide, which covers sink-by-sink output handling, and focuses on the decision between the two interfaces.

Key takeaways

• Short answer: any model output that triggers an action should be structured and validated against a strict schema; free text belongs only where a human reads the result or where the output is treated purely as data, never as an instruction to execute.
• Schemas stop malformed actions, not malicious ones. A call such as send_email(to="[email protected]", body=<customer list>) is perfectly valid JSON. Shape validation must be followed by semantic validation and authorisation outside the model.
• The weakest field defines the risk. One free-form string that reaches a shell, a query or a URL turns a structured call back into free text at the point that matters, as the Gemini CLI allowlist bypass showed.
• There is no parameterisation for prompts. The NCSC notes that SQL injection can be mitigated because parameterised queries separate data from instructions, and LLMs have no equivalent separation. Structured output narrows the action space; it does not separate data from instructions inside the model.
• Authorisation never lives in the schema. OWASP LLM06 calls for complete mediation: downstream systems decide what is allowed, using the identity of the user on whose behalf the agent acts.

The two interfaces in plain terms

Free text: the model writes prose, a command line, a query or a code block, and application code extracts the action from it with regular expressions, keyword matching or a second parsing step. The action is whatever the parser finds. Typical examples are agents that emit a shell command in a fenced block, text-to-SQL features that execute the model's query, and planners that write "Next step: call refund for order 1234" for a parser to pick up.

Structured output: the model emits a JSON object that conforms to a declared schema, either because the model provider constrains generation to the schema (often called structured outputs, JSON mode or strict tool calling) or because the application validates the object after generation and rejects anything that does not conform. Tool calling is the common form: the tool definition carries a JSON Schema for its arguments, and the model returns a tool name plus an arguments object. In MCP, each tool's inputSchema is a JSON Schema that defaults to the 2020-12 dialect and must be a valid schema object; servers must validate all tool inputs, and clients should validate structured results against a tool's outputSchema.

Two different things share the word "structured". Constrained decoding controls what the model can generate. Schema validation controls what the application accepts. The MCP specification even notes that its structuredContent field is server-produced result data and unrelated to schema-constrained model generation. A secure design uses validation on every action regardless of whether generation was constrained, because the constraint is a convenience that the application does not control.

Side-by-side comparison

Threat model: free text actions

Free text gives the model, and anyone who can influence the model, the full grammar of the target interpreter. OWASP LLM05 lists the results: cross-site scripting, server-side request forgery, privilege escalation and remote code execution when output reaches downstream components without validation, and SQL injection when model-generated queries run without parameterisation.

• Interpreter injection: a model asked for one command can produce two. Any parser that inspects only the start of the string, or that splits on the wrong delimiter, executes what it did not check (ATT&CK T1059 Command and Scripting Interpreter). In the Gemini CLI Prompt Injection Flaw 2025, hidden instructions in a README led the agent to run a command that began with an allowlisted grep followed by a semicolon and a second command, which sent the developer's environment variables to a remote server; the allowlist appears to have judged the string by its leading program name.
• Parser confusion: extraction logic written with regular expressions disagrees with the real interpreter on quoting, escapes, Unicode and whitespace, so the reviewed action differs from the executed one.
• Exfiltration through rendering: free text rendered as Markdown or HTML can carry image URLs and links that leak data when displayed, which is why the LLM05 guide treats rendering as a sink in its own right.
• No contract to test against: because there is no declared shape, there is nothing to assert in unit tests except "the parser found something", and regressions arrive silently when a model update changes phrasing.

Threat model: structured actions

Structured output removes parser ambiguity and the open grammar, but the model still chooses which tool to call and with which values. Anything that can steer the model, including indirect prompt injection in retrieved documents, tool results and emails (ATLAS AML.T0051.001), can steer those choices within the schema.

• Valid but malicious arguments: the main failure mode. The schema says to is a string in email format, and an injected instruction supplies an attacker's address. The call passes validation and is executed as designed (ATLAS AML.T0086 Exfiltration via AI Agent Tool Invocation).
• Valid but harmful tool choice: the model calls delete_records instead of list_records. Both are declared; the schema cannot tell which the user intended (ATLAS AML.T0101 Data Destruction via AI Agent Tool Invocation).
• Free text smuggled inside a field: a query, command, path, url or template string carries a full interpreter grammar. A structured call with a raw SQL field is free text with a JSON wrapper.
• Loose schemas: JSON Schema does not reject unknown properties unless told to. The OWASP Input Validation Cheat Sheet points out that listing a property alone neither requires it nor rejects unknown fields; additionalProperties: false and required must be set explicitly. The 2020-12 specification also says format is collected as an annotation by default, and its assertion behaviour must be disabled unless the format-assertion vocabulary is enabled, so "format": "email" on its own may validate nothing.
• Unanchored patterns: the 2020-12 validation vocabulary states that regular expressions in pattern are not implicitly anchored. A pattern written as [a-z0-9-]+ without ^ and $ matches any string that contains one such character.
• Untrusted tool output fed back: structured actions often consume structured results. A tool result that carries instructions in a text field is read by the model as context. In the Sentry MCP Agentjacking 2026 case, a planted error report returned through an MCP tool told coding agents to run a package command, which they did with the developer's privileges.

What schemas do and do not stop

The validation and authorisation stack

The OWASP LLM Prompt Injection Prevention Cheat Sheet says to validate proposed tool arguments and enforce the caller's permissions in execution code outside the model, and states that LLMs do not enforce a boundary between instructions and data in a prompt. A practical stack has five layers, each deterministic and each outside the model.

1. Shape: validate every call against a closed schema: additionalProperties: false, every field required or explicitly optional, enum for any fixed vocabulary, maxLength on strings, minimum and maximum on numbers, anchored pattern values, and maxItems on arrays. Reject, do not repair.
2. Semantics: check that values make sense for the operation, the distinction the OWASP Input Validation Cheat Sheet draws between syntax and semantics. Recipients belong to an approved domain or the user's contacts, paths resolve inside the workspace after canonicalisation, URLs resolve to allowed hosts, amounts fall within the transaction limit.
3. Safe execution: never pass a field to an interpreter as a string. Build commands as argument arrays without a shell, use parameterised queries, encode for the rendering context. This is where LLM05 controls apply to each field.
4. Authorisation: the downstream system decides, using the end user's delegated identity and scopes, whether this caller may perform this action on this resource. OWASP LLM06 describes this as complete mediation, with authorisation implemented in downstream systems rather than relying on the model. See the AI Agent Authorisation Guide.
5. Intent and approval: for irreversible or high-impact actions, require human confirmation of the exact validated call, displayed field by field. The cheat sheet warns that approval fatigue weakens review, so reserve it for actions that need it. See the AI Agent Action Approvals and Spending Limits Guide.

The NCSC adds a design principle that sits above all five: an LLM processing content from a party should drop to that party's privilege level, so an agent reading external email should not hold privileged tools at the same time.

Designing schemas that carry security weight

• Replace verbs in strings with tools: instead of run(command: string), expose list_files(dir), read_file(path) and run_tests(suite: enum). Each narrow tool gets its own authorisation and approval policy.
• Replace query strings with filters: instead of sql: string, accept table: enum, filters: [{field: enum, op: enum, value}] and limit: integer, maximum 500, and build the parameterised query in code.
• Use identifiers, not free values: a recipient_id from the user's contact list is safer than a free email address; a record_id checked against the caller's entitlements is safer than a name.
• Bound everything: string lengths, array sizes, numeric ranges and date windows. Bounds turn a bulk export into a validation error.
• Keep schemas versioned and reviewed: a schema change that widens a field is a security change. In MCP, tool definitions come from the server and clients must treat tool annotations as untrusted unless they come from trusted servers, so pin and diff tool definitions. See MCP Tool Poisoning Detection.

Decision criteria

Small teams should start by moving every action-triggering output to strict tool schemas and parameterised execution, which removes the interpreter class of bug for little cost. Larger organisations should add a shared policy layer that applies semantic checks and authorisation consistently across agents, instead of each team writing its own validator. The comparison of where such checks run is in Pre-Execution Policy Checks vs Post-Execution Monitoring for Agents.

Worked scenario

A finance team builds an agent that reads supplier emails and raises payment requests. Version one asked the model for free text such as "Pay invoice INV-2207 to supplier Acme, 4,200 GBP" and parsed it with regular expressions. A test email with an injected line produced "Pay invoice INV-2207 to account 12-34-56 87654321", which the parser accepted.

Version two moved to a create_payment_request tool with a strict schema. The test still succeeded: the model filled payee_account with the attacker's details, and the call was valid. Shape validation had fixed parsing, not the attack.

Version three removed payee_account entirely and replaced it with supplier_id, looked up against the supplier master file; payment details come only from that record. amount is bounded by the invoice total from the accounts payable system; invoice_id must exist and be unpaid. The agent's identity can create requests but not approve them, and any request above a threshold or for a supplier whose bank details changed in the last 30 days goes to a human approver who sees the validated fields. The injected email now produces either a validation error or a request routed to the correct supplier.

The lesson generalises: move trust-critical values out of what the model writes and into what the system looks up.

How to implement

1. Inventory action paths: list every place model output is executed, sent, stored or rendered, and which interface it uses. See the MCP vs Native Function Calling guide for how tool calls reach code.
2. Convert free text actions: replace parsed text with tool schemas, starting with paths that write or send.
3. Tighten schemas: close objects, add bounds and enums, anchor patterns, and enable format assertion only where your validator supports it.
4. Add semantic and authorisation checks in the executor, in audit mode first: log what would be rejected for two weeks, review false positives, then enforce. Keep a feature flag to return to audit mode as the rollback path.
5. Test adversarially: run indirect injection cases that try to change recipients, paths, amounts and tool choice, and assert rejection. Include chained commands and unanchored pattern cases. See the AI security regression testing guide.

Detection and measurement

Log every proposed call with tool name, arguments, validation outcome per layer, the identity used and the content sources in context. Alert on repeated rejections from one session, which the NCSC notes can indicate an attacker refining an attack, and on approved calls whose arguments include values not seen in the user's request.

• Percentage of action-triggering outputs that pass through a strict schema (target 100).
• Number of tool fields typed as unbounded free-form strings that reach an interpreter (target zero).
• Semantic and authorisation rejection rate per tool, and its trend after model changes.
• Adversarial test pass rate for injection cases per release.
• Share of high-impact actions that required and received human confirmation.

Common failure patterns

• Calling schema validation "injection protection": it removes malformed calls and leaves every valid malicious one.
• Keeping a raw command or sql field: the most dangerous field is the one that still accepts a grammar.
• Relying on constrained decoding alone: generation constraints vary by provider and model; the executor must validate independently.
• Trusting format and pattern blindly: unasserted formats and unanchored patterns pass almost anything.
• Letting the model supply identity or entitlement: a user_id or role field filled by the model is an authorisation bypass waiting to happen.

Framework mapping

Practitioner checklist

• No model output reaches a shell, query engine, eval or template as an unvalidated string.
• Every action-triggering output is a tool call validated against a closed schema at the executor.
• Schemas set additionalProperties: false, explicit required, bounds and anchored patterns.
• Free-form command and query fields have been replaced by narrow tools, enums and typed filters.
• Trust-critical values (payees, recipients, identities) are looked up by the system, not written by the model.
• Downstream systems authorise each call with the end user's delegated identity.
• Privileged tools are withheld while the agent processes untrusted content.
• High-impact actions require confirmation of the exact validated call.
• Tool definitions are pinned and diffed; schema widening goes through security review.
• Adversarial injection tests run on every model or prompt change.

Standards and references

• OWASP: LLM05:2025 Improper Output Handling
• OWASP: LLM06:2025 Excessive Agency
• OWASP Cheat Sheet Series: LLM Prompt Injection Prevention Cheat Sheet
• OWASP Cheat Sheet Series: Input Validation Cheat Sheet
• JSON Schema: Validation Vocabulary, draft 2020-12
• Model Context Protocol: Tools (revision 2026-07-28)
• NCSC: Prompt injection is not SQL injection

Related NHI Mgmt Group resources: OWASP LLM Top 10 LLM05 Improper Output Handling Guide · Prompt Injection Guide · OWASP LLM Top 10 LLM06 Excessive Agency Guide · MCP vs Native Function Calling · Streamable HTTP vs stdio MCP Transport