AI Platform Account Takeover: Prevent and Respond

A stolen login or session cookie for ChatGPT, Claude, Gemini or a similar assistant gives an attacker months of conversation history, uploaded files and, increasingly, live connectors into email, drives and code. This is a different problem from LLMjacking, where stolen API keys are used to run up model usage: here the target is the human user's account and everything it can see. Infostealer malware, credential reuse and personal accounts used for work make AI platform accounts an easy, high-value catch. This guide is for identity, SaaS security and SOC teams who need to decide which controls to put around AI assistant accounts, how to detect a takeover and how to respond when one happens.

Key takeaways

The account is a data store and a gateway. Chat history, projects, uploaded files, memory and connector grants mean one takeover can expose source code, contracts and downstream SaaS data.
Infostealers already harvest AI logins at scale. Group-IB reported 101,134 stealer-infected devices with saved ChatGPT credentials in the year to May 2023, and session cookies are taken alongside passwords.
Personal accounts are the blind spot. Work data in consumer accounts sits outside your SSO, logging and offboarding; domain capture and SSO enforcement are the controls that bring it back.
IdP session controls may not reach the AI app. Many AI platforms keep their own session cookies, so sign-in frequency and token binding at your identity provider do not automatically end an attacker's session.
Response means revoking sessions and grants, not just resetting passwords. Log out all devices, revoke connector OAuth tokens, review exports and shared links, then rotate any secrets pasted into chats.

What an attacker gets from an AI platform account

Treat an enterprise AI assistant account as a privileged SaaS account. The value lies less in the model than in what users have put into it and connected to it.

Asset in the account	Why it matters	Typical exposure
Conversation history	Users paste code, logs, customer data, credentials and strategy into chats	Bulk reading or data export of everything retained
Projects, files and knowledge bases	Shared reference material, often more curated than email	Download of uploaded documents; visibility changes to share them out
Memory and custom instructions	Persistent personal and business context	Profiling of the user for follow-on phishing
Connectors and apps (email, drive, calendar, code hosting, ticketing)	Delegated OAuth access into other systems	The attacker can ask the assistant to search or act across those systems as the user
Agents, custom GPTs and scheduled tasks	Automations that run with the user's grants	Persistence: a modified agent or instruction keeps working after a password reset
Admin roles	Workspace settings, SSO, retention, compliance exports	Disabling controls or exporting the whole workspace

The connector row is where this differs most from older SaaS takeover. The attacker does not need to hold the downstream OAuth token themselves: an authenticated chat session can drive the connector. The Vercel Context.ai OAuth breach shows the reverse path too: an employee granted a consumer AI app full access to a corporate Google account, the AI vendor was breached, and the stolen OAuth token was used to take over the Workspace account. For API key abuse, which is a separate threat, see the LLMjacking Guide.

How AI platform accounts are taken over

Route	ATT&CK reference	What makes it work	Primary control
Infostealer harvests saved passwords	T1555.003 Credentials from Web Browsers	Password saved in the browser, no MFA, credential never rotated	SSO with phishing-resistant MFA; stealer log monitoring
Infostealer or malicious extension steals session cookies	T1539 Steal Web Session Cookie, T1550.004 Web Session Cookie	Long-lived app session cookie that is not bound to the device	Shorter app session lifetimes; managed browsers; extension allowlists
Credential stuffing and password reuse	T1078.004 Cloud Accounts	Personal or email-and-password accounts outside SSO	Domain capture and SSO enforcement
Phishing for the AI login or the IdP login	T1078.004 Cloud Accounts	Push or OTP MFA that can be relayed	Passkeys or FIDO2 security keys at the IdP
Theft of a downstream OAuth grant from the AI vendor	T1528 Steal Application Access Token	Broad, long-lived grants to AI apps, including unsanctioned ones	OAuth app governance, scope limits and expiry
Email-based sign-in links or codes	T1078.004 Cloud Accounts	Magic link or emailed code where the mailbox is already compromised	Enforce SSO so email login is not available for corporate domains

The Group-IB figures above came from its own threat intelligence platform and covered June 2022 to May 2023; they are dated, but they show that stealer families were already capturing AI platform credentials as a matter of course. For how stealer data is traded and reused, see the Infostealer Logs and Session Token Abuse Guide.

Bring accounts under corporate identity

The single most effective control is to make sure work use happens in managed workspaces that authenticate through your identity provider.

Verify your domains and enforce SSO: Anthropic documents that Claude SSO enforcement is applied by email domain, so members on an enforced domain, including the Primary Owner, must authenticate through SSO. Without enforcement, users can keep signing in with email links, Google or Apple, which bypasses your MFA policy.
Provision and deprovision automatically: SCIM removes a user when they are removed from the IdP application. JIT provisioning is weaker on the leaver side: Anthropic notes that JIT never removes members automatically, and an unassigned user remains a member, holding a seat, until an admin removes them. Pair JIT with a scheduled reconciliation. See the Joiner-Mover-Leaver Guide.
Do not rely on app-level MFA for the workforce: OpenAI documents that ChatGPT account MFA uses an authenticator app with recovery and email code fallbacks, and that MFA cannot currently be enforced for an entire workspace. App-level MFA is a good personal control, but workforce assurance has to come from SSO and IdP policy.
Use phishing-resistant MFA at the IdP: passkeys or FIDO2 keys on the IdP sign-in remove the relayable OTP and push routes.
Find the personal accounts: use SaaS discovery, DNS and proxy logs and expense data to find consumer AI subscriptions on corporate email, then invite them into the managed workspace or close them. See the Shadow AI Discovery Guide.

Session controls: where your IdP stops

Session theft defeats MFA because the attacker replays a cookie issued after authentication. The difficulty with AI platforms is that the session usually belongs to the AI vendor's application, not your IdP.

Control	What it covers	Limit to plan around
Microsoft Entra sign-in frequency	Reauthentication for Microsoft and non-Microsoft SAML, OAuth2 and OIDC apps	Microsoft documents that apps keeping their own cookies and not redirecting back to Entra may not have it enforced
Microsoft Entra Token Protection	Device-bound tokens so stolen tokens fail from another device	Supported resources are listed Microsoft services such as Exchange Online, SharePoint Online and Teams, not third-party AI apps
Google Workspace web session duration	Reauthentication for Google web services (default 14 days)	Google documents that OAuth-authenticated apps and native mobile apps are not enforced
AI platform "log out of all devices"	Ends the vendor-side sessions for one account	OpenAI notes that turning on MFA does not end existing sessions, so this must be a separate step
Managed or enterprise browser	Restricts extensions, profile sync and cookie export on managed devices	Does not help on unmanaged personal devices

The practical answer is layered: shorten the AI application's own session lifetime where the vendor allows it, restrict access to managed devices through the vendor's network or device controls where offered, block unmanaged extensions, and treat infostealer detection on any device that has touched the AI workspace as a session revocation trigger. The OWASP Session Management Cheat Sheet sets out the server-side expectations to test vendors against: session renewal on authentication, idle and absolute timeouts, and server-side invalidation on logout. See the Session Hijacking and Cookie Theft Guide.

Govern connectors and agents as delegated access

Allowlist connectors at workspace level: enable only the connectors you have reviewed, and scope them to groups that need them. Each enabled connector is an OAuth grant held on behalf of the user and should be in your non-human identity inventory.
Prefer read-only scopes: a connector that can send email, push code or modify tickets turns a takeover into an action, not only a disclosure.
Review grants to AI apps in your IdP and SaaS platforms: list third-party OAuth apps with access to Google Workspace, Microsoft 365, GitHub and Slack, flag AI vendors, and remove unsanctioned or dormant ones.
Track agents and scheduled tasks: custom assistants, instructions and scheduled actions survive password changes. Include them in takeover response.

Detecting a takeover

Detection depends on getting the AI platform's logs into your SIEM. On the enterprise tiers, OpenAI's Compliance Platform for ChatGPT Enterprise and Edu exposes audit, authentication and app log categories through the Compliance API with a workspace Admin key. Anthropic's Claude audit logs, available on Enterprise organisations, record sign-ins by method (SSO, Google, Apple, magic link, phone code), SSO enforcement changes, conversation and project activity, file uploads and data export start and completion; an export covers the past 180 days.

Sign-in method drift: a magic link, Google or Apple sign-in for a user on an SSO-enforced domain, or SSO enforcement being switched off.
Impossible or new access context: AI platform activity from an IP, ASN or country that does not match the user's IdP sign-ins in the same period, which is a strong cookie replay signal.
Bulk collection: data export started, many conversations opened or files downloaded in a short window, or project visibility changed to shared.
Connector activity spikes: unusual search volume through mail or drive connectors, or connectors newly enabled by a user who has not used them before.
Admin changes: retention, SSO, domain or role changes outside change control.
Endpoint signal: an infostealer detection on a device where the user is signed in to an AI workspace.

Metrics

Percentage of AI platform users on corporate domains authenticating through enforced SSO.
Number of personal AI accounts on corporate email found and migrated or closed per quarter.
Percentage of enabled connectors with a named owner and reviewed scopes.
Time from infostealer detection to AI platform session revocation.
Percentage of AI workspaces whose audit logs reach the SIEM.

Responding to a suspected takeover

Contain sessions: revoke the user's IdP sessions and refresh tokens, then use the AI platform's log-out-everywhere or admin session revocation; suspend the account if activity is ongoing.
Revoke delegated access: disconnect connectors from the user's AI account and revoke the matching OAuth grants in Google Workspace, Microsoft 365 or other platforms.
Check for persistence: review custom instructions, memory, custom assistants, scheduled tasks, shared links, project membership and any API keys created.
Scope exposure: pull audit logs for exports, file downloads and conversation access during the window; identify secrets, personal data and regulated data in those conversations.
Rotate what was pasted: any credential, token or key that appears in chat history must be rotated, not just the AI account password. See the Infostealer Logs Monitoring and Response Guide.
Clean the endpoint: if a stealer was involved, reimage the device and assume every browser-stored credential and session on it is compromised.
Assess notification duties: personal data in chats or connected systems may trigger breach notification obligations; involve privacy and legal early.

Choosing controls by organisation size

Situation	Minimum viable control set	Trade-off
Small team on a Team or Business plan	SSO with enforced domain if the plan offers it, IdP MFA with passkeys, JIT plus a monthly member review, connectors off by default	Audit logs and compliance APIs are often limited to enterprise tiers, so detection relies on IdP and endpoint signals
Mid-size organisation with one sanctioned assistant	Enterprise tier with SCIM, audit log export to the SIEM, connector allowlist, block consumer AI domains for corporate identities at the proxy	Blocking consumer tools without offering a sanctioned one drives use to personal devices
Large organisation with several AI platforms	Standard identity baseline applied to every platform, central OAuth grant inventory, session revocation automation tied to EDR and stealer log alerts	Each vendor exposes different log fields and session controls, so normalisation work is substantial

Whatever the size, the decision that matters most is whether work use is allowed in personal accounts. If it is, none of the controls above apply to that data; if it is not, the policy needs a sanctioned alternative and discovery to enforce it. See the Shadow AI Policy Template.

Common failure patterns

Resetting the password and stopping there: existing sessions, connector grants and modified agents survive a password change.
SSO configured but not enforced: users keep their old email or social sign-in, and that is the credential in the stealer log.
JIT provisioning without a leaver process: departed staff remain workspace members with full history until someone removes them.
Assuming IdP session policy applies: the AI app's own cookie outlives the IdP session, so a replayed cookie still works.
Ungoverned AI app OAuth grants: as in the Vercel Context.ai OAuth breach, a broad grant to an AI tool becomes the attacker's route when that vendor is breached.
Credentials without MFA and never rotated: the Snowflake breach showed how infostealer credentials, some years old, keep working against SaaS platforms that do not enforce MFA.

Framework mapping

Framework	Reference	Relevance
NIST SP 800-53 Rev. 5	IA-2, IA-11 Re-authentication, AC-12 Session Termination, AC-2 Account Management	SSO and MFA, session lifetime, provisioning and removal
NIST CSF 2.0	PR.AA-03; DE.CM-03	Authentication of users and services; monitoring of personnel activity
MITRE ATT&CK	T1539, T1550.004, T1555.003, T1528, T1078.004	Cookie and credential theft, token theft, cloud account abuse
ISO/IEC 27001:2022 Annex A	A.5.18 Access rights; A.8.5 Secure authentication	Provisioning, review and removal; strong authentication

Practitioner checklist

Verify corporate domains on every AI platform in use and enforce SSO for them.
Use SCIM where available; if JIT is the only option, reconcile members against the IdP weekly.
Require phishing-resistant MFA at the IdP for AI workspace access.
Find and migrate or close personal AI accounts on corporate email.
Allowlist connectors, prefer read-only scopes and record each grant with an owner.
Stream AI platform audit and authentication logs to the SIEM and alert on sign-in method drift, exports and admin changes.
Make infostealer detection on a device trigger IdP and AI platform session revocation.
Write an AI account takeover runbook covering sessions, connectors, agents, exports and secret rotation.

AI Platform Account Takeover Guide: Protecting Chat Histories, Sessions and Connected Data