What breaks when AI agents have to interpret page structure instead of using declared tools?

Why This Matters for Security Teams

When an AI agent has to interpret page structure, the security problem is no longer just access. It is also misclassification of intent, state, and authority. A link, button, or section header can look harmless to a human reviewer while the agent treats it as an instruction path. That is why declared tools matter: they constrain the action space and make purpose explicit instead of inferred.

This risk is already showing up in agentic environments. NHIMG research on OWASP NHI Top 10 and the AI Agents: The New Attack Surface report shows that organisations are deploying agents faster than they are governing their actions. The governance gap is especially dangerous when agents are asked to "read" interfaces instead of calling declared tools, because the page becomes both input and execution surface. Current guidance from the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework points toward constrained, observable interaction patterns rather than open-ended page scraping. In practice, many security teams encounter the failure only after an agent has already clicked the wrong element, consumed the wrong context, or acted on a stale page state.

How It Works in Practice

Declared tools reduce ambiguity by giving the agent a bounded interface: a named action, defined inputs, typed outputs, and a clear audit trail. That is materially different from asking the model to infer meaning from DOM structure, visible text, or layout. With a declared tool, the control plane decides what the agent may do; with page interpretation, the model is effectively guessing which element is authoritative.

Security teams usually implement this by separating read and act paths. The agent can inspect content through a controlled retrieval layer, but actions such as submit, delete, approve, or navigate require explicit tool calls. That makes it easier to apply policy checks, log intent, and revoke access if behavior drifts. It also aligns with the direction of CSA MAESTRO agentic AI threat modeling framework, which treats tool boundaries, authorization, and misuse paths as first-class design concerns. NHIMG’s Analysis of Claude Code Security reinforces the same lesson: tool boundaries are a control, not just a convenience.

• Use declared tools for each action class, not generic "browser control" for everything.
• Limit page-reading agents to observation, then require a separate tool for state-changing actions.
• Attach policy checks at tool invocation time, not after the page interaction has already happened.
• Log the declared intent, the parameters supplied, and the response returned for review.

Where this guidance breaks down is in highly dynamic documentation portals, single-page apps, or sites with unstable selectors and frequent layout changes, because the agent may lose its reference point even when the tool is well designed.

Common Variations and Edge Cases

Tighter tool boundaries often increase integration overhead, requiring organisations to balance reliability against development effort and UX flexibility. That tradeoff is real, especially when legacy portals were never designed for programmatic access.

There is no universal standard for this yet, but current best practice is evolving toward hybrid patterns: declared tools for writes, constrained retrieval for reads, and explicit confirmation steps before any irreversible action. This matters most when an agent must move across pages, compare section context, or preserve workflow state. Without those constraints, the model may treat navigation chrome as content, content as controls, or a previously valid element as still authoritative after the page changes. NHIMG’s coverage of the Gemini AI Breach - Google Calendar Prompt Injection and the CoPhish OAuth Token Theft via Copilot Studio show how quickly trust assumptions collapse when agents over-interpret interface cues. The practical lesson is simple: if the interface can change meaning based on layout, localisation, or page state, the agent needs a declared tool, not more prompting.

Related resources from NHI Mgmt Group

• What breaks when AI agents discover tools at runtime instead of using hardcoded lists?
• What breaks when AI agents trust MCP tools after a single approval?
• How should security teams govern AI agents that call APIs instead of using a UI?
• What breaks when AI is bolted onto existing applications instead of using AI-first architecture?