Enterprise AI platforms hand a model the reach of a user, an application or an agent, so every weakness in permissions, credentials and logging becomes something a prompt can find or trigger. Most organisations now run several platforms at once: a productivity copilot grounded on mail and documents, a chat assistant with connectors, a cloud model service behind internal applications, a coding assistant in the IDE and a model hub for open weights. Each has its own admin console, identity model and logging, and each changes monthly. This page is the overview of the whole AI Platforms category: what the field covers, the concepts that apply to every platform, the main threats, controls by maturity, who owns what, and the frameworks that apply. It then links every NHI Mgmt Group platform guide for the product-level detail. It is written for security architects, identity teams, cloud and data protection practitioners and the owners of AI platforms.
Key takeaways
- Platforms inherit access, they rarely create it. Copilots ground answers on what the user can already open, so oversharing and stale group membership are the first thing to fix, as the Microsoft 365 Copilot and Gemini guides both show.
- Every platform runs on non-human identities. API keys, service roles, OAuth apps, connector tokens and agent identities do the real work, and stolen keys are the root of LLMjacking.
- Logging is often off or partial by default. Amazon Bedrock model invocation logging must be switched on, and audit coverage differs by product tier, so verify what is recorded before go-live.
- Guardrails are a layer, not a boundary. Content filters and prompt classifiers were bypassed in EchoLeak and the Gemini calendar case, so least privilege on data and tools has to carry the weight.
- Govern platforms as a portfolio. One inventory, one approval path, one logging standard and one set of data rules across SaaS assistants, cloud model services and self-hosted models beats a separate policy per product.
What the field covers
The AI Platforms category covers the commercial and open platforms through which an organisation consumes or builds AI, and the security settings each one exposes. It is about the platform layer, not about the model's internal behaviour (that is the AI Threats and Model Security category) or about autonomous agents as a class (the agent categories).
| Platform family | Typical examples | Who uses it | Main security question |
|---|---|---|---|
| Productivity copilots | Microsoft 365 Copilot, Gemini for Google Workspace | All staff | What can the assistant ground on, and is that content correctly permissioned and labelled? |
| Enterprise chat assistants | ChatGPT Enterprise, Claude for Enterprise | Knowledge workers | Which connectors, data retention settings and workspace controls apply, and who administers them? |
| Cloud model services | Amazon Bedrock, Azure OpenAI and Azure AI Foundry, Google Vertex AI | Developers and application teams | Which roles can invoke which models, are guardrails enforced, and is invocation logged? |
| Coding assistants | GitHub Copilot | Engineers | What code and secrets reach the model, and how are suggestions and agent actions reviewed? |
| Model hubs and open weights | Hugging Face, self-hosted open-weight models | ML engineers | Are downloaded artefacts safe to load, and are access tokens and private repositories protected? |
| Automation and low-code AI | Workflow tools with AI steps | Makers and business teams | Which shared integration identities do AI steps use, and who approved them? |
Core concepts
- Shared responsibility: the provider secures the model hosting and service plane; the customer owns identities, data exposure, configuration, connectors and logging. Contract terms on training use and retention vary by tier and must be read for each product. See the Shared Responsibility Models Guide.
- Grounding and retrieval: copilots and RAG applications retrieve content at question time and pass it to the model. The retrieval layer decides what can leak, which is why permission-aware retrieval matters. See the Permission-Aware RAG Guide.
- Delegated versus application access: a connector that acts with the user's delegated OAuth token can only reach what that user can; a connector using an application permission or shared service account reaches everything it was granted, for every user.
- Model access entitlement: in cloud model services, the right to call a model is an IAM permission (for example
bedrock:InvokeModelon a foundation-model ARN). Allowlisting approved models by explicit deny is more reliable than relying on any enablement step. - Guardrails and content safety: provider filters for harmful content, prompt attack detection and sensitive data redaction. In Bedrock they apply only when a request names a guardrail unless enforcement is configured; the
bedrock:GuardrailIdentifiercondition key makes them mandatory. - Agents and actions: platforms increasingly let assistants call tools, send mail and change records. Once an assistant can write, prompt injection becomes a data movement and integrity risk, not only a disclosure risk.
- Gateways: an AI gateway or LLM gateway sits between applications and model providers to centralise keys, quotas, logging and policy. See the AI Gateway and LLM Gateway Guide.
Main threats
The OWASP Top 10 for LLM Applications 2025 is the common vocabulary for application-level risk, and MITRE ATLAS gives technique IDs for detection and threat modelling. The table maps the threats that recur across platforms.
| Threat | How it shows up on platforms | OWASP LLM 2025 | MITRE ATLAS / ATT&CK |
|---|---|---|---|
| Oversharing surfaced by the assistant | Copilot answers from "everyone" shared sites and stale memberships | LLM02 Sensitive Information Disclosure | AML.T0057 LLM Data Leakage |
| Indirect prompt injection | Crafted email, invite or document steers the assistant (EchoLeak, Gemini calendar) | LLM01 Prompt Injection | AML.T0051.001 Indirect |
| Stolen API keys and cloud credentials | Attackers invoke models on the victim's bill (LLMjacking) | LLM10 Unbounded Consumption | AML.T0055 Unsecured Credentials, AML.T0034 Cost Harvesting, T1078.004 Cloud Accounts |
| Over-privileged connectors and agents | Application-permission connector or agent service role reaches far more than the use case | LLM06 Excessive Agency | AML.T0053 AI Agent Tool Invocation |
| Supply chain compromise | Malicious model files, poisoned SDKs or gateway packages (LiteLLM) | LLM03 Supply Chain | AML.T0010 AI Supply Chain Compromise, T1195.002 |
| System prompt and configuration leakage | Instructions, keys or internal URLs in system prompts are extracted | LLM07 System Prompt Leakage | AML.T0056 Extract LLM System Prompt |
| Unsafe output handling | Rendered Markdown, links or generated code fetch attacker URLs or execute | LLM05 Improper Output Handling | AML.T0025 Exfiltration via Cyber Means |
| Secrets in code and prompts | Keys pasted into chats or committed code suggested back | LLM02 | T1552.001 Credentials In Files |
Three incidents illustrate the pattern. In EchoLeak (CVE-2025-32711) one crafted email could make Microsoft 365 Copilot place internal data in a link that the browser fetched; the classifier, link redaction and content security policy were each bypassed. In the Gemini calendar prompt injection a calendar invite made the assistant write a summary of private meetings into a new event. In LLMjacking, stolen cloud credentials and API keys were used to call Bedrock, Azure OpenAI and other services, with Sysdig estimating potential costs above $46,000 per day. For the threat side in depth, see the AI Threats and Model Security Guide and the Prompt Injection Guide.
Choosing a platform model
Most organisations end up with several of these. The decision is which workloads go where, not which single platform wins.
| Option | Choose it when | Security trade-off |
|---|---|---|
| Productivity copilot in your existing suite | Value comes from mail, documents and meetings already in that suite | Inherits every permission problem in the tenant; data stays inside the suite's compliance boundary and admin tooling |
| Enterprise chat assistant | Staff need a general assistant with selected connectors | Another admin plane and SSO integration; connector scope and retention settings must be governed separately |
| Cloud model service in your cloud account | You build applications or agents and want IAM, network and key control | Most control, most responsibility: you own roles, guardrail enforcement, logging and quotas |
| Self-hosted or open-weight model | Data residency, cost or customisation requires it | Full stack is yours, including endpoint authentication, patching and model file safety. See the Self-Hosted LLM Security Guide. |
Small organisations: standardise on one suite copilot and one chat assistant, turn on SSO and audit logging, restrict connectors to an approved list and block personal accounts at the proxy. Large organisations: add a model gateway for all application traffic, enforce model allowlists by organisation policy (for example AWS SCPs or Azure Policy), separate builder and logging-admin roles, and run platform onboarding through a formal AI governance review. See the AI Governance Guide.
Key controls by maturity
| Area | Foundational | Managed | Advanced |
|---|---|---|---|
| Inventory | List of approved platforms, owners and licence counts | Inventory of connectors, agents, API keys and service roles per platform | Continuous discovery of shadow AI and unapproved model calls; see the Shadow AI Discovery Guide |
| Identity | SSO and MFA for every console; no shared admin accounts | Least-privilege roles, separate builder, operator and logging-admin duties | Short-lived credentials and workload identity federation in place of static keys |
| Data exposure | Oversharing clean-up before rollout; sensitive sites excluded | Sensitivity labels and DLP applied to AI interactions | Permission-aware retrieval for every custom RAG application |
| Model access | Only approved models callable | Explicit deny on unapproved model identifiers at organisation level | Gateway-enforced routing, quotas and per-application keys |
| Guardrails | Provider content filters on | Guardrails mandatory by policy condition, prompt attack detection enabled | Custom evaluations and red teaming before each major release |
| Actions and agents | Write actions disabled or limited to pilots | Human confirmation for sending, sharing and changing records | Per-agent identities with scoped tokens and kill switches |
| Logging | Admin audit logs retained and sent to SIEM | Prompt and response or invocation logging where lawful | Detections for injection signals, spend anomalies and logging changes |
Rollout approach
- Inventory and classify: list every platform in use, including trials and personal accounts seen in proxy logs, and assign an owner and data classification to each.
- Prepare data: fix oversharing, remove "everyone" grants on sensitive sites, apply labels, and decide which repositories are excluded from grounding.
- Harden identity: enforce SSO, MFA and conditional access on consoles; replace long-lived API keys with roles where the platform allows; store remaining keys in a secrets manager. See the LLMjacking Guide.
- Turn on logging first: enable audit and invocation logging and confirm events arrive in the SIEM before any user is licensed.
- Pilot ring: license a small group (often IT, security and a willing business unit) with connectors and write actions limited; review what the assistant surfaced and tune exclusions.
- Expand in rings: widen access department by department, enabling connectors and agent features only after their owner and scope are approved.
- Roll back cleanly: keep a documented path to remove licences, disable connectors and agent features, revoke API keys and service-role trust, and restrict sites from search, so a problem can be contained in hours rather than weeks.
Roles and ownership
| Role | Owns |
|---|---|
| AI platform owner | Configuration, licensing, feature enablement, change log and roll-back plan for one platform |
| Identity and access team | SSO, admin roles, OAuth app consent, service accounts, API key lifecycle and access reviews |
| Data protection and records | Labels, DLP, retention, data residency and the lawful basis for prompt logging |
| Cloud security | IAM policies, organisation guardrails, network paths and key management for cloud model services |
| Application and ML engineering | RAG pipelines, system prompts, output handling, model selection and dependency hygiene |
| SOC and detection engineering | Log onboarding, detections and AI incident playbooks; see the AI Agent Incident Response Playbook |
| AI governance or risk committee | Approval of new platforms and high-risk use cases, policy and exceptions |
Where identity fits
Every AI platform is an identity system in disguise. Copilots act as delegated identities for the user; connectors and agents act through OAuth apps, service principals and service roles; applications call models with API keys. The Bedrock service roles for agents and knowledge bases, Azure managed identities, Vertex AI service accounts, Hugging Face access tokens and GitHub Copilot's organisation policies are all machine identities that need an owner, least privilege, rotation and review. The LiteLLM PyPI breach shows the consequence: a poisoned CI scanner exposed a publishing token, and the malicious package harvested LLM provider API keys alongside cloud credentials. See the Agent Identity and Access Guide.
Detection and measurement
- Log sources: suite audit logs for copilot interactions (Microsoft 365 records prompt injection and jailbreak detections as audit properties), cloud control plane logs (CloudTrail, Azure Activity Log, Cloud Audit Logs), model invocation logs, gateway logs and OAuth consent events.
- High-value alerts: changes to logging configuration, first use of a model not on the allowlist, new API keys or long-lived keys, spend spikes, new connectors or agents published, and AI-generated links pointing outside the organisation.
| Metric | What good looks like |
|---|---|
| Platforms with named owner and logging verified | 100% of approved platforms |
| Long-lived AI API keys in use | Falling each quarter; none older than the rotation policy |
| Sensitive sites or repositories exposed to organisation-wide grounding | Zero after remediation, tracked weekly during rollout |
| Connectors and agents with approved scope and owner | 100%, reviewed at least quarterly |
| Model calls through the gateway or approved endpoints | Above 95% of application traffic |
| Mean time to revoke a leaked AI key | Hours, not days |
Common failure patterns
- Licensing before permissions: rolling a copilot out to everyone, then discovering it answers from years of overshared content.
- Static keys in code and CI: provider keys in repositories, environment variables and CI secrets readable by every job, the route used in LLMjacking and the LiteLLM compromise.
- Advisory guardrails: filters configured but not enforced, so any application can call the model without them.
- Logging assumed, not verified: invocation logging off, or audit events on a tier the organisation did not buy, discovered only during an incident.
- Connectors with application permissions: a shared integration identity that lets every user's prompt reach every mailbox or record.
- External content treated as trusted: inbound email, invites and shared documents grounded and acted on without confirmation, as in EchoLeak and the Gemini calendar case.
Standards and frameworks
| Framework | Where AI platforms fit |
|---|---|
| NIST AI RMF 1.0 (January 2023) and NIST AI 600-1 Generative AI Profile (July 2024) | Govern, Map, Measure and Manage functions for AI risk; the profile adds generative AI risks such as information security and data privacy. See the NIST AI RMF Guide and NIST AI 600-1 Generative AI Profile Guide. |
| NIST CSF 2.0 | ID.AM-02 inventories of software and services; PR.AA-05 access permissions with least privilege; DE.CM-09 monitoring of software and runtime environments; GV.SC-07 supplier risk |
| NIST SP 800-53 Rev. 5 | AC-6 Least Privilege, AU-2 Event Logging, CM-8 System Component Inventory, SI-4 System Monitoring, SR-3 Supply Chain Controls and Processes, RA-3 Risk Assessment |
| ISO/IEC 42001:2023 | AI management system requirements for organisations that provide or use AI; platform onboarding and supplier controls sit within it |
| NCSC and partners: Guidelines for secure AI system development (November 2023) | Secure design, development, deployment, and operation and maintenance, including logging and update management |
| OWASP Top 10 for LLM Applications 2025 | Application risk vocabulary for anything built on the platforms; see the OWASP Top 10 for LLM Applications Guide |
| MITRE ATLAS | Technique IDs such as AML.T0051 LLM Prompt Injection and AML.T0034 Cost Harvesting for threat models and detections |
| ENISA Multilayer Framework for Good Cybersecurity Practices for AI (June 2023) | Cybersecurity foundations, AI-specific and sector-specific layers |
Regulatory obligations such as the EU AI Act sit alongside these frameworks; see the EU AI Act Compliance Guide.
Where to go next (all guides in this category)
Vendor-neutral foundations
- Enterprise AI Copilot Security Guide: the risk model for any assistant that searches company data and acts on it, with rollout and monitoring steps.
Productivity copilots
- Microsoft 365 Copilot Security Deep Dive: oversharing, Restricted Content Discovery, Purview labels and DLP, agents and audit events.
- Gemini for Google Workspace Security Guide: admin controls, data protection and how Gemini uses Workspace permissions.
Enterprise chat assistants
- ChatGPT Enterprise Security Guide: workspace admin controls, data handling and connectors.
- Claude for Enterprise Security Guide: SSO and provisioning, admin settings, data handling and integrations.
Cloud model services
- Amazon Bedrock Security Guide: IAM for models, agents and knowledge bases, guardrail enforcement, API keys and invocation logging.
- Azure OpenAI and Azure AI Foundry Security Guide: Entra ID roles, managed identities, API keys, network isolation and logging.
- Google Vertex AI Security Guide: service accounts, VPC Service Controls, agents and data protection on Google Cloud.
Developer and model platforms
- GitHub Copilot Security Guide: organisation policies, content exclusion, coding agent controls and audit.
- Hugging Face Security Guide: model and dataset safety, Spaces, organisation settings and access tokens.
Related pillars: the AI Threats and Model Security Guide for attacks on models and data, the LLM Security Guide for application security, and the Agent Controls and Monitoring Guide for agents built on these platforms.
Practitioner checklist
- Keep one inventory of every AI platform, connector, agent, API key and service role, each with a named owner.
- Remediate oversharing and apply sensitivity labels before licensing a copilot beyond a pilot ring.
- Enforce SSO and MFA on every AI admin console and separate builder, operator and logging-admin roles.
- Replace long-lived AI API keys with roles or short-lived credentials; vault and rotate any that remain.
- Allowlist approved models by explicit deny at organisation level and make guardrails mandatory by policy.
- Turn on audit and invocation logging, confirm it reaches the SIEM, and alert on any change to it.
- Require human confirmation for assistant actions that send, share or change data, especially with external content in context.
- Restrict connectors to delegated access where possible and review application-permission integrations quarterly.
- Set budgets and spend alerts on every model service to catch key abuse early.
- Document a tested roll-back for each platform: licences, connectors, agent features, keys and trust policies.
Standards and references
- NIST: AI Risk Management Framework
- NCSC: Guidelines for secure AI system development
- ISO: ISO/IEC 42001:2023 Artificial intelligence management system
- OWASP: Top 10 for LLM Applications 2025
- MITRE ATLAS: LLM Prompt Injection
- NIST: Cybersecurity Framework 2.0
- NIST: SP 800-53 Rev. 5
- ENISA: Multilayer Framework for Good Cybersecurity Practices for AI
Related NHI Mgmt Group resources: AI Threats and Model Security Guide · Enterprise AI Copilot Security Guide · LLMjacking Guide · AI Gateway and LLM Gateway Guide · Agent Identity and Access Guide