Securing Enterprise AI Copilots and Assistants

Enterprise AI assistants, such as the copilots built into productivity suites, CRM, IT service management and developer platforms, work with the same permissions as the person using them, and that turns years of hidden over-sharing into easily discovered data. In principle, permission-trimmed answers are safe. In practice, most organisations have files shared with everyone, broad group memberships and forgotten access, and an assistant that can search everything a user can technically reach finds it in seconds. Copilots are also gaining the ability to act: sending messages, updating records, calling connectors and running custom agents, each of which is a non-human identity with its own access. This guide covers the identity and data risks of enterprise copilots and how to prepare, deploy, govern and monitor them. It is for identity, data protection and collaboration platform teams, and for security leads approving a rollout.

Key takeaways

• Existing over-sharing is the biggest risk: copilots generally respect permissions, so they surface whatever users can already reach. Find and fix broad sharing before a broad rollout.
• Labels, scoped discovery and DLP keep the crown jewels out: use sensitivity labels, temporary exclusion of sensitive sites and data loss prevention for AI interactions while clean-up continues.
• Connectors and custom agents are NHIs: each needs an owner, scoped permissions, a consent policy and a place in your inventory, or it becomes the easiest way around user-level controls.
• Prompt injection turns content into commands: EchoLeak, the Gemini calendar invite and ForcedLeak all used content the assistant read to make it leak data. Limit what assistants can do with untrusted input and where output can go.
• Log and review AI interactions: which content was referenced, which agents and plugins ran, and which actions were taken, then remove agents and connectors nobody uses.

How enterprise copilots access data

Most enterprise copilots ground their answers in organisational data through an index or graph, then pass the retrieved content to a large language model as context. Microsoft states that its Copilot "only surfaces organizational data to which individual users have at least view permissions", and that its Semantic Index "honors the user identity-based access boundary". Google says Gemini in Workspace follows existing Workspace permissions: "If you don't have permission to see a file, the AI can't see it or use it either." Those guarantees are only as good as the permissions underneath them.

Agent-building features let users create custom agents with their own connectors and actions. See the Low-Code Agent Platforms Guide.

The risks

• Over-shared content surfaced: files shared with "everyone", company-wide links, broad sites and old project spaces become answers to simple questions. Microsoft's own deployment guidance tells administrators to find sites with "oversized audiences", "Everyone except external users" (EEEU) usage, broken inheritance and inactive or ownerless status.
• Stale access: movers who kept access to old departments can now search that content effortlessly. See the Joiner-Mover-Leaver Guide.
• Connector over-reach: OWASP describes "excessive permissions" where an extension built to act for one user "accesses downstream systems with a generic high-privileged identity".
• Prompt injection to data leak: in EchoLeak 2025, according to BleepingComputer's account of Aim Labs' research, a crafted email retrieved by Microsoft 365 Copilot's RAG engine made it place internal data in a link sent to the attacker; Microsoft fixed CVE-2025-32711 server-side. In the Gemini AI Calendar Prompt Injection 2026, Miggo showed a calendar invite description that made Gemini summarise a user's private meetings into a new event the attacker could see.
• Agents as phishing infrastructure: in CoPhish 2025, Datadog Security Labs showed that a Copilot Studio agent's "Login" button could deliver an OAuth consent attack from a genuine Microsoft domain and send the resulting token to an attacker-controlled URL.
• Exfiltration through allowed destinations: in ForcedLeak 2025, Noma Labs found that instructions planted in a Web-to-Lead form could make Salesforce Agentforce send CRM data to an expired domain still on an allowlist; Salesforce patched it and re-secured the domain.
• Data in prompts and outputs: sensitive content pasted into unmanaged AI tools, or AI output stored where it should not be. After the Samsung ChatGPT Source Code Leak 2023, TechCrunch reported that Samsung temporarily restricted generative AI tools on company devices.

OWASP's Top 10 for LLM Applications 2025 frames these as LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure and LLM06 Excessive Agency. NIST AI 600-1 adds that generative AI "expands the available attack surface, as GAI itself is vulnerable to attacks like prompt injection or data poisoning".

Before rollout: prepare permissions

1. Find over-sharing: content shared with everyone or large groups, anonymous and company-wide links, broken inheritance, and inactive or ownerless sites with broad access. Data access governance and posture reports in the collaboration platform are the usual starting point.
2. Prioritise sensitive locations: HR, legal, finance, mergers and acquisitions, executive, security and customer data, plus any site holding regulated personal data.
3. Apply interim protection: exclude high-risk sites from AI discovery while they are reviewed. Microsoft's Restricted Content Discovery is an example; it is "designed as a temporary governance control" and "doesn't change existing permissions", so users with access can still open the content directly.
4. Clean up: remove broad sharing, rescope links to named groups, fix group membership, archive inactive sites and assign accountable site owners.
5. Label: apply sensitivity labels and encryption to the most sensitive content. Microsoft states that for content encrypted by its information protection service, Copilot "honors the usage rights granted to the user". See the Data Classification Guide.
6. Remove interim protection only when the underlying permissions are fixed, and record the decision.

This is access governance applied urgently. See the Access Reviews Guide and the Permission-Aware RAG Guide.

During rollout

• Pilot with known access: start with groups whose permissions are well understood and expand in waves, checking audit data between waves.
• Approve connectors and agents centrally: check whether each uses delegated or shared identities, what it can reach and who owns it. Microsoft notes that admins "have full control to select which agents are allowed" in the integrated apps settings.
• Control agent building and sharing: restrict who can create and share custom agents, require an owner and a description of the data and actions involved, and review before wide sharing. See the Agentic AI Identity Guide.
• Turn on DLP for AI: where supported, block AI processing of prompts containing sensitive information types, stop labelled files and emails being used in responses, and stop sensitive prompts going to external web search. See the Generative AI Data Loss Prevention Guide.
• Consider external content: some platforms can exclude email from external domains as grounding data, which reduces the indirect prompt injection surface.
• Set acceptable use and train users: what may be pasted into assistants, which tools are sanctioned and how to spot manipulated output. See the AI Acceptable Use Policy Template.

When copilots act

OWASP traces excessive agency to "excessive functionality; excessive permissions; excessive autonomy" and recommends executing extensions "in user's context" with the minimum privileges necessary. Tier actions by impact and match the control to the tier.

• Make the confirmation show what will happen, not a model-written summary that injected content could falsify.
• Limit actions for users handling sensitive processes such as payments, HR cases and legal matters.
• Restrict where output can be sent: block rendering of external images and links in responses where the platform allows, and keep allowlisted domains current, as ForcedLeak's expired domain showed. See the AI Agent Authorisation Guide.

Connectors and custom agents as non-human identities

Every connector, plugin and custom agent holds a credential or token and acts on data, so treat it like any other NHI. OWASP's agentic Top 10 warns that "without a distinct, governed identity of its own, an agent operates in an attribution gap that makes enforcing true least privilege impossible" (ASI03 Identity and Privilege Abuse).

• Inventory: list every connector, plugin and published agent with its owner, data sources, actions, identity type and audience. See the Shadow AI Discovery Guide.
• Consent policy: Datadog's CoPhish research notes that by default all Entra ID member users can register applications and that privileged application administrators are not bound by user consent policies. Restrict app registration, tighten consent and monitor consent grants. See the Entra ID Applications and Consent Guide and the SaaS and OAuth App Governance Guide.
• Prefer delegated, scoped access over shared integration identities with tenant-wide read rights.
• Lifecycle: review agents and connectors quarterly, remove unused ones and transfer ownership when makers leave. See the Identity Governance for AI Agents Guide.

Monitoring

Audit logs for AI interactions are the evidence base for both detection and governance. Microsoft logs Copilot interactions under the CopilotInteraction record type, including an AccessedResources field listing referenced files with their sensitivity label, and fields for the agent and plugins involved.

• Who asked what: collect interaction records with the user, application, agent and resources referenced, and keep them long enough to investigate.
• Sensitive content reach: alert when labelled or high-risk content is referenced by users outside the expected group, or when one user's AI queries touch unusually many sensitive files.
• Agent behaviour: alert on agent actions outside normal patterns, new connectors, and new or changed agents published to wide audiences.
• Consent and app events: new consent grants to high-risk delegated permissions, and new application registrations.
• Housekeeping: review and remove unused agents and connectors. See the AI Agent Observability and Incident Response Guide.

Framework mapping

Indicative alignment prepared by NHI Mgmt Group, not an official mapping by any of these bodies. Control IDs were checked against NIST CSWP 29, the SP 800-53 Rev. 5 catalogue, the CIS Controls v8.1 list, a published ISO/IEC 27001:2022 Annex A list and the OWASP LLM and Agentic Top 10 texts.

For AI risk governance as a whole, the NIST AI Risk Management Framework organises activities under Govern, Map, Measure and Manage, and NIST AI 600-1 profiles generative AI risks including data privacy and information security. See the AI Governance Guide.

Common failure patterns

• Licences first, permissions later: broad rollout before any over-sharing assessment, followed by an urgent clean-up after a user finds salary or board papers in an answer.
• Interim exclusions that become permanent: sensitive sites are hidden from AI discovery and never remediated, so the underlying exposure remains for direct access and search.
• Labels without coverage: a labelling scheme exists but most sensitive content is unlabelled, so label-based DLP and AI restrictions catch little.
• Uncontrolled agent building: any user can publish an agent with their own access to the whole organisation, with no owner review, as CoPhish showed is exploitable for phishing.
• Treating injection as a filter problem: relying on a classifier alone, when EchoLeak and the Gemini invite were phrased as ordinary business text; Miggo notes Google "has already deployed a separate language model to detect malicious prompts, and yet the path still existed".
• Stale allowlists: trusted domains for links and images are not reviewed, leaving an exfiltration channel, as in ForcedLeak.
• No logs until an incident: AI interaction auditing is not collected or retained, so nobody can say what an assistant referenced or did.

Metrics that show progress

Implementation roadmap

Practitioner checklist

• Run an over-sharing assessment and fix the most sensitive locations before broad rollout.
• Use temporary discovery exclusions only as a bridge to real permission fixes.
• Apply sensitivity labels and restrict AI processing of the most sensitive content.
• Turn on DLP for AI prompts, labelled content and web search where supported.
• Roll out in waves and check audit data between waves.
• Approve connectors centrally and prefer delegated, scoped access over shared identities.
• Restrict who can build and share agents; give every agent an owner and a review date.
• Tighten application registration and consent policies, and monitor consent grants.
• Require explicit approval for external, irreversible and financial actions.
• Collect and retain AI interaction logs, alert on sensitive content reach, and remove unused agents and connectors.

Standards and references

• OWASP GenAI Security Project: LLM01:2025 Prompt Injection
• OWASP GenAI Security Project: LLM02:2025 Sensitive Information Disclosure
• OWASP GenAI Security Project: LLM06:2025 Excessive Agency
• OWASP GenAI Security Project: Top 10 for Agentic Applications for 2026
• NIST: AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0)
• NIST: AI 600-1, Generative Artificial Intelligence Profile
• Microsoft Learn: Data, Privacy, and Security for Microsoft Copilot
• Microsoft Learn: Restrict discovery of SharePoint sites and content
• Microsoft Learn: Audit logs for Copilot and AI applications
• Google Workspace Admin Help: Generative AI in Google Workspace Privacy Hub
• Datadog Security Labs: CoPhish, using Microsoft Copilot Studio as a wrapper for OAuth phishing
• Miggo Security: Weaponizing Calendar Invites, a semantic attack on Google Gemini

Related NHI Mgmt Group resources: Permission-Aware RAG Guide · Low-Code Agent Platforms Guide · Shadow AI Discovery Guide · Microsoft 365 Security Guide