TL;DR: Parseable JSON is not the same as a trusted machine contract, TruFoundry argues: structured outputs still require schema enforcement, business validation, and authorization before any downstream action. That boundary matters because tool calls, provider-specific schema limits, and retry loops can turn a formatting problem into an access-control problem.
At a glance
What this is: TruFoundry argues that structured outputs are only useful as a production control when schema enforcement, business validation, and authorization all sit between the model and the action.
Why it matters: This matters because IAM, NHI, and agentic AI teams must treat typed model outputs and tool calls as governed contracts, not trustworthy intent, before any external side effect is allowed.
Context
Structured output turns a model response into a typed interface, but it does not turn language generation into truth. The governance gap appears when software treats parseable JSON or a schema-conforming object as enough proof to execute an action, especially where tool calls, API payloads, and agent workflows touch identity-bound permissions.
In practice, the control boundary has to sit after generation as well as before execution. That is where validation, policy checks, and authorization separate machine-readable structure from a request that is actually allowed to change state. For AI agents and other non-human identities, this is the point where interface design becomes access control.
Key questions
Q: What breaks when structured outputs are treated as trusted machine contracts?
A: Teams lose the separation between syntax, business truth, and authority. A schema-valid object can still contain the wrong value, target the wrong tenant, or trigger an action that should be denied. The fix is to treat generation as untrusted input and enforce validation and authorization after the model responds.
Q: Why do schema-valid tool calls still create security risk?
A: Because a tool call describes intent, not permission. If the system accepts the arguments and executes the action without a separate policy decision, the model has effectively crossed into privileged execution. Security teams should keep authorization at the runtime boundary, where scope and approvals are actually enforceable.
Q: What are the signs that structured output validation is failing?
A: Common signs include repeated repair loops, downstream rejection of supposedly valid objects, mismatches between provider conformance and application rules, and state-changing requests that arrive before checks are complete. Those symptoms show the workflow is trusting structural correctness too early.
Q: How should teams govern AI systems that can take actions as well as generate outputs?
A: Treat the agent as a governed actor, not just a model output stream. Require action-level logging, tool-call traceability, authorization boundaries, and approval gates before the system can write to records or invoke downstream tools. If an AI system can change state, its authority must be scoped, monitored, and revocable like any other privileged non-human identity.
Technical breakdown
Why valid JSON is not a trusted contract
JSON mode only guarantees parseability, not correctness. A model can still emit the wrong enum, omit a required field, or return an object shape that downstream code cannot safely consume. Schema-constrained generation improves this by narrowing the allowed structure, but the guarantee depends on the provider path, supported schema vocabulary, and whether the response was refused or truncated. That means the interface is still probabilistic at the generation layer and only becomes reliable when application code checks the result against the contract it actually needs.
Practical implication: validate model output against the exact contract your workflow depends on, not just against JSON syntax.
Where business validation has to sit after schema checks
A structurally valid object can still be semantically wrong. The schema may accept an amount, date, or customer identifier that violates tenant ownership, freshness, policy limits, or local business rules. That is why application validation belongs after generation and after schema enforcement. Deterministic checks should handle facts the schema cannot know, such as whether a resource exists, whether a timestamp is current enough, or whether the request crosses an environment boundary. In agentic systems, this is often the difference between safe automation and an unintended side effect.
Practical implication: place deterministic business rules between structured output and any database lookup, workflow step, or external API call.
How tool calls turn structured output into access control
Tool calling is structured output with consequence. The typed tool name and arguments define an intended action, but they do not grant the authority to execute it. That is why authorization has to be separate from validation: a tool call can be well-formed, policy-compliant in shape, and still forbidden because the caller lacks scope, approval, or environment-level permission. For AI agents, this is an identity problem as much as an output problem, because the agent is effectively acting as a non-human principal with constrained rights.
Practical implication: enforce runtime authorization and approval policy at the tool boundary, not inside the prompt or schema alone.
Threat narrative
Attacker objective: The attacker objective is to convert untrusted model output into an unauthorized machine action.
- Entry occurs when a model produces a structurally valid object that downstream software mistakes for trusted intent.
- Credential or privilege abuse follows when a tool call or API payload is accepted without separate authorization checks.
- Impact appears when the validated-looking object triggers a state-changing action, data access, or deployment step that should have been blocked.
Breaches seen in the wild
- Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.
- DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Schema-valid is not security-valid: the most dangerous failure mode in structured AI workflows is assuming that typed output proves intent, ownership, or permission. The article is right to separate JSON shape from business truth, because those are different control problems with different owners. Practitioners should treat schema enforcement as interface hygiene and keep security decisions in downstream policy.
Tool calls collapse the distance between suggestion and action: once the model can name an operation and supply arguments, the control question shifts from output quality to authority. That makes agentic systems a non-human identity problem as well as an application design problem, because the model is acting through a governed principal with scope limits. Practitioners should design for that identity boundary explicitly.
Repair loops create governance debt if they are unbounded: retries can improve usability, but repeated correction cycles can also hide failing rules and delay escalation. The safer pattern is a machine-readable defect response with a bounded retry budget, so deterministic validation remains the source of truth. Practitioners should make failure paths as explicit as success paths.
Portable interfaces do not mean portable guarantees: a common response surface across providers can mask different schema subsets, truncation behaviour, and enforcement semantics. That matters for governance because a contract that works in one model path may fail differently in another. Practitioners should test the exact provider and model combination they intend to ship, then version the schema like any other API contract.
From our research library:
- Software supply chain attacks were projected to cost organisations $60 billion in 2025.
- Read next: AI Agent Observability, Audit and Incident Response Guide
What this signals
Contract pressure is moving from prompt design to enforcement design: once models can emit tool-ready objects, teams need controls that distinguish representation from permission. The practical shift is toward smaller model-facing schemas, deterministic validation, and explicit authorization at execution time.
Agentic workflows expose a governance gap in the identity stack: the moment a model can trigger external actions, it behaves like a non-human principal that needs scoped rights, auditable approvals, and revocation paths. Identity teams should expect structured outputs to become part of access control design, not just application plumbing.
The organisation's analysis aligns with the wider warning that AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
For practitioners
- Define a three-stage contract boundary Separate parseable structure, business-valid content, and authorized action into distinct checkpoints so a model never jumps straight from generation to execution.
- Enforce deterministic post-generation validation Use Pydantic, JSON Schema, database lookups, and policy checks after generation to verify facts that the schema cannot prove, including ownership, freshness, and tenant scope.
- Gate every tool call with runtime authorization Treat the tool name and argument object as intent only, then require explicit permission, approval, or policy evaluation before any external side effect occurs.
- Version schemas like application APIs Track schema versions alongside traces, test old and new contracts against representative traffic, and use shadow validation before promoting a new response shape.
- Bound repair loops and escalate failures Return machine-readable defects, allow only a small number of targeted retries, and escalate once the same invariant fails repeatedly instead of looping indefinitely.
Key takeaways
- Structured outputs reduce ambiguity, but they do not prove that a model response is truthful, allowed, or safe to execute.
- The critical control gap is the handoff from typed output to business validation and runtime authorization.
- For AI agents and other non-human identities, schema design, validation, and permissioning have to be governed as one workflow.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Non-Human Identity Top 10, OWASP Agentic AI Top 10 and OWASP API Security Top 10 address the attack and risk surface, while NIST AI RMF and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Non-Human Identity Top 10 | NHI-04 — Insecure Authentication | Structured tool calls need separate authority checks before any action executes. |
| Recommendation — Require runtime authorization before a model-generated tool call can execute. | ||
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | The article centers on non-human principals acting with scoped rights and approval boundaries. |
| Recommendation — Constrain agent permissions so typed outputs cannot exceed granted privilege. | ||
| OWASP API Security Top 10 | API5 — Broken Function Level Authorization | Tool calls are function-like actions that must be denied unless explicitly allowed. |
| Recommendation — Enforce function-level authorization on every model-triggered action. | ||
| NIST AI RMF | MANAGE — AI risk management and response | The article is about governing AI output risk across validation and action boundaries. |
| Recommendation — Embed validation, authorization, and failure handling into AI system governance. | ||
| NIST SP 800-53 Rev 5 | AC-6 — Least Privilege | Model-triggered actions should only execute with minimal required authority. |
| Recommendation — Apply least privilege to every tool and workflow the model can invoke. | ||
Key terms
- Structured Output: Structured output is a response format that follows a predefined shape, such as a table, checklist, schema, or file tree. It makes machine and human review easier because the answer is predictable, parseable, and less likely to drift into unbounded prose.
- Business Validation: Business validation is the deterministic checking of model output against real-world rules such as ownership, freshness, policy limits, and tenant boundaries. It sits after schema validation because a syntactically valid object can still be operationally wrong or unsafe.
- Tool-Call Authorisation: Tool-call authorisation is the practice of making each agent action request a distinct security decision. Instead of granting broad access to the agent process, the system evaluates the specific tool, inputs, role, and context before execution, which creates a reviewable control boundary.
- Machine Contract: A machine contract is the explicit technical agreement a system exposes to automated consumers through documentation, schemas, error responses, and UI structure. It defines how a non-human actor is expected to behave, and it becomes a security boundary when access depends on strict, repeatable interpretation.
Deepen your knowledge
The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, IAM, and secrets management. It helps security teams design controls for non-human principals that act through tools and typed contracts.
Published by the NHIMG editorial team on August 21, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org