What breaks when prompt injection and over-sharing are not controlled in copilot deployments?

Why This Matters for Security Teams

Copilot deployments fail in a predictable way when prompt injection and over-sharing are treated as content problems instead of control problems. The risk is not only that an assistant answers too much. It is that the assistant may retrieve, summarize, or act on data outside the user’s intended boundary, especially when connectors, chat history, and delegated permissions are all in play. OWASP’s OWASP Agentic AI Top 10 treats injection and excessive autonomy as core threats, not edge cases.

NHIMG research shows the same pattern in real incidents. The CoPhish OAuth Token Theft via Copilot Studio case illustrates how a seemingly helpful assistant can become a delivery path for credential theft when trust boundaries are weak. The practical failure is usually quiet: a copilot surfaces a document, forwards a message, or exposes a token because the system trusted the prompt, the connector, or the user too much. In practice, many security teams encounter this only after sensitive data has already been retrieved or shared, rather than through intentional testing.

How It Works in Practice

Prompt injection succeeds when an attacker places instructions inside content the copilot is allowed to read, such as email, documents, tickets, chat messages, or web pages. If the assistant follows those instructions, it may ignore policy, call tools, or reveal context that should have stayed hidden. Over-sharing compounds the problem because the copilot is often given broad read access to make the experience feel useful. That means the assistant can see more than the user should see, and then reproduce that material in its output.

Security teams should think in terms of data minimisation, tool scoping, and output control. A safer design limits what the copilot can retrieve, what it can send back, and which actions require explicit confirmation. Current guidance suggests using policy filters at retrieval time and output time, not just at the UI layer. For agentic deployments, the real control question is whether the system can distinguish user intent from injected instructions inside untrusted content.

• Restrict connector scope to the minimum data set needed for the task.
• Separate user-visible content from hidden system instructions and tool prompts.
• Apply runtime policy checks before retrieval, summarisation, and action execution.
• Log prompt, tool, and data-access events so leakage paths can be reconstructed.

NHIMG’s Ultimate Guide to NHIs is relevant here because copilot connectors and service identities frequently carry excessive privileges, and 97% of NHIs carry excessive privileges according to NHIMG research. That matters when a copilot is using delegated access to mailboxes, document stores, or ticketing systems. The assistant is not just reading data; it is operating through an identity that can be over-permissioned, poorly rotated, or insufficiently monitored. These controls tend to break down when the copilot can chain multiple connectors across environments because the blast radius expands faster than review processes can keep up.

Common Variations and Edge Cases

Tighter filtering often increases friction, requiring organisations to balance user convenience against containment. That tradeoff becomes sharper in deployments that span multiple tenants, federated repositories, or regulated datasets, where a useful answer may legitimately require partial access to sensitive material.

There is no universal standard for this yet, but best practice is evolving toward layered controls: prompt sanitisation, connector allowlists, data loss prevention, and human approval for high-impact actions. The OWASP Agentic Applications Top 10 and NHIMG’s Gemini AI Breach — Google Calendar Prompt Injection show why indirect prompts and content injection are especially dangerous when assistants can query live business systems. For some organisations, the harder problem is not retrieval but disclosure through summarisation, where the copilot unintentionally aggregates fragments that were never meant to be combined.

In high-risk environments, current guidance suggests treating every external input as untrusted and every outbound response as a possible exfiltration path. The exception is tightly bounded, offline, or read-only deployments, where the attack surface is smaller and the acceptable risk is clearer. Even then, monitoring must detect unusual retrieval volume, connector abuse, and repeated attempts to coerce tool use. Without that, prompt injection and over-sharing turn a copilot into a data router instead of a controlled assistant.

Related resources from NHI Mgmt Group

• What breaks when prompt injection is not controlled in agentic workflows?
• What breaks when indirect prompt injection is not controlled in AI systems?
• What breaks when prompt injection is not controlled in MCP environments?
• What breaks when prompt injection or poisoned project context is not controlled in AI-assisted development?