A stolen login or session cookie for ChatGPT, Claude, Gemini or a similar assistant gives an attacker months of conversation history, uploaded files and, increasingly, live connectors into email, drives and code. This is a different problem from LLMjacking, where stolen API keys are used to run up model usage: here the target is the human user's account and everything it can see. Infostealer malware, credential reuse and personal accounts used for work make AI platform accounts an easy, high-value catch. This guide is for identity, SaaS security and SOC teams who need to decide which controls to put around AI assistant accounts, how to detect a takeover and how to respond when one happens.
Key takeaways
- The account is a data store and a gateway. Chat history, projects, uploaded files, memory and connector grants mean one takeover can expose source code, contracts and downstream SaaS data.
- Infostealers already harvest AI logins at scale. Group-IB reported 101,134 stealer-infected devices with saved ChatGPT credentials in the year to May 2023, and session cookies are taken alongside passwords.
- Personal accounts are the blind spot. Work data in consumer accounts sits outside your SSO, logging and offboarding; domain capture and SSO enforcement are the controls that bring it back.
- IdP session controls may not reach the AI app. Many AI platforms keep their own session cookies, so sign-in frequency and token binding at your identity provider do not automatically end an attacker's session.
- Response means revoking sessions and grants, not just resetting passwords. Log out all devices, revoke connector OAuth tokens, review exports and shared links, then rotate any secrets pasted into chats.
What an attacker gets from an AI platform account
Treat an enterprise AI assistant account as a privileged SaaS account. The value lies less in the model than in what users have put into it and connected to it.
| Asset in the account | Why it matters | Typical exposure |
|---|---|---|
| Conversation history | Users paste code, logs, customer data, credentials and strategy into chats | Bulk reading or data export of everything retained |
| Projects, files and knowledge bases | Shared reference material, often more curated than email | Download of uploaded documents; visibility changes to share them out |
| Memory and custom instructions | Persistent personal and business context | Profiling of the user for follow-on phishing |
| Connectors and apps (email, drive, calendar, code hosting, ticketing) | Delegated OAuth access into other systems | The attacker can ask the assistant to search or act across those systems as the user |
| Agents, custom GPTs and scheduled tasks | Automations that run with the user's grants | Persistence: a modified agent or instruction keeps working after a password reset |
| Admin roles | Workspace settings, SSO, retention, compliance exports | Disabling controls or exporting the whole workspace |
The connector row is where this differs most from older SaaS takeover. The attacker does not need to hold the downstream OAuth token themselves: an authenticated chat session can drive the connector. The Vercel Context.ai OAuth breach shows the reverse path too: an employee granted a consumer AI app full access to a corporate Google account, the AI vendor was breached, and the stolen OAuth token was used to take over the Workspace account. For API key abuse, which is a separate threat, see the LLMjacking Guide.
How AI platform accounts are taken over
| Route | ATT&CK reference | What makes it work | Primary control |
|---|---|---|---|
| Infostealer harvests saved passwords | T1555.003 Credentials from Web Browsers | Password saved in the browser, no MFA, credential never rotated | SSO with phishing-resistant MFA; stealer log monitoring |
| Infostealer or malicious extension steals session cookies | T1539 Steal Web Session Cookie, T1550.004 Web Session Cookie | Long-lived app session cookie that is not bound to the device | Shorter app session lifetimes; managed browsers; extension allowlists |
| Credential stuffing and password reuse | T1078.004 Cloud Accounts | Personal or email-and-password accounts outside SSO | Domain capture and SSO enforcement |
| Phishing for the AI login or the IdP login | T1078.004 Cloud Accounts | Push or OTP MFA that can be relayed | Passkeys or FIDO2 security keys at the IdP |
| Theft of a downstream OAuth grant from the AI vendor | T1528 Steal Application Access Token | Broad, long-lived grants to AI apps, including unsanctioned ones | OAuth app governance, scope limits and expiry |
| Email-based sign-in links or codes | T1078.004 Cloud Accounts | Magic link or emailed code where the mailbox is already compromised | Enforce SSO so email login is not available for corporate domains |
The Group-IB figures above came from its own threat intelligence platform and covered June 2022 to May 2023; they are dated, but they show that stealer families were already capturing AI platform credentials as a matter of course. For how stealer data is traded and reused, see the Infostealer Logs and Session Token Abuse Guide.
Bring accounts under corporate identity
The single most effective control is to make sure work use happens in managed workspaces that authenticate through your identity provider.
- Verify your domains and enforce SSO: Anthropic documents that Claude SSO enforcement is applied by email domain, so members on an enforced domain, including the Primary Owner, must authenticate through SSO. Without enforcement, users can keep signing in with email links, Google or Apple, which bypasses your MFA policy.
- Provision and deprovision automatically: SCIM removes a user when they are removed from the IdP application. JIT provisioning is weaker on the leaver side: Anthropic notes that JIT never removes members automatically, and an unassigned user remains a member, holding a seat, until an admin removes them. Pair JIT with a scheduled reconciliation. See the Joiner-Mover-Leaver Guide.
- Do not rely on app-level MFA for the workforce: OpenAI documents that ChatGPT account MFA uses an authenticator app with recovery and email code fallbacks, and that MFA cannot currently be enforced for an entire workspace. App-level MFA is a good personal control, but workforce assurance has to come from SSO and IdP policy.
- Use phishing-resistant MFA at the IdP: passkeys or FIDO2 keys on the IdP sign-in remove the relayable OTP and push routes.
- Find the personal accounts: use SaaS discovery, DNS and proxy logs and expense data to find consumer AI subscriptions on corporate email, then invite them into the managed workspace or close them. See the Shadow AI Discovery Guide.
Session controls: where your IdP stops
Session theft defeats MFA because the attacker replays a cookie issued after authentication. The difficulty with AI platforms is that the session usually belongs to the AI vendor's application, not your IdP.
| Control | What it covers | Limit to plan around |
|---|---|---|
| Microsoft Entra sign-in frequency | Reauthentication for Microsoft and non-Microsoft SAML, OAuth2 and OIDC apps | Microsoft documents that apps keeping their own cookies and not redirecting back to Entra may not have it enforced |
| Microsoft Entra Token Protection | Device-bound tokens so stolen tokens fail from another device | Supported resources are listed Microsoft services such as Exchange Online, SharePoint Online and Teams, not third-party AI apps |
| Google Workspace web session duration | Reauthentication for Google web services (default 14 days) | Google documents that OAuth-authenticated apps and native mobile apps are not enforced |
| AI platform "log out of all devices" | Ends the vendor-side sessions for one account | OpenAI notes that turning on MFA does not end existing sessions, so this must be a separate step |
| Managed or enterprise browser | Restricts extensions, profile sync and cookie export on managed devices | Does not help on unmanaged personal devices |
The practical answer is layered: shorten the AI application's own session lifetime where the vendor allows it, restrict access to managed devices through the vendor's network or device controls where offered, block unmanaged extensions, and treat infostealer detection on any device that has touched the AI workspace as a session revocation trigger. The OWASP Session Management Cheat Sheet sets out the server-side expectations to test vendors against: session renewal on authentication, idle and absolute timeouts, and server-side invalidation on logout. See the Session Hijacking and Cookie Theft Guide.
Govern connectors and agents as delegated access
- Allowlist connectors at workspace level: enable only the connectors you have reviewed, and scope them to groups that need them. Each enabled connector is an OAuth grant held on behalf of the user and should be in your non-human identity inventory.
- Prefer read-only scopes: a connector that can send email, push code or modify tickets turns a takeover into an action, not only a disclosure.
- Review grants to AI apps in your IdP and SaaS platforms: list third-party OAuth apps with access to Google Workspace, Microsoft 365, GitHub and Slack, flag AI vendors, and remove unsanctioned or dormant ones.
- Track agents and scheduled tasks: custom assistants, instructions and scheduled actions survive password changes. Include them in takeover response.
Detecting a takeover
Detection depends on getting the AI platform's logs into your SIEM. On the enterprise tiers, OpenAI's Compliance Platform for ChatGPT Enterprise and Edu exposes audit, authentication and app log categories through the Compliance API with a workspace Admin key. Anthropic's Claude audit logs, available on Enterprise organisations, record sign-ins by method (SSO, Google, Apple, magic link, phone code), SSO enforcement changes, conversation and project activity, file uploads and data export start and completion; an export covers the past 180 days.
- Sign-in method drift: a magic link, Google or Apple sign-in for a user on an SSO-enforced domain, or SSO enforcement being switched off.
- Impossible or new access context: AI platform activity from an IP, ASN or country that does not match the user's IdP sign-ins in the same period, which is a strong cookie replay signal.
- Bulk collection: data export started, many conversations opened or files downloaded in a short window, or project visibility changed to shared.
- Connector activity spikes: unusual search volume through mail or drive connectors, or connectors newly enabled by a user who has not used them before.
- Admin changes: retention, SSO, domain or role changes outside change control.
- Endpoint signal: an infostealer detection on a device where the user is signed in to an AI workspace.
Metrics
- Percentage of AI platform users on corporate domains authenticating through enforced SSO.
- Number of personal AI accounts on corporate email found and migrated or closed per quarter.
- Percentage of enabled connectors with a named owner and reviewed scopes.
- Time from infostealer detection to AI platform session revocation.
- Percentage of AI workspaces whose audit logs reach the SIEM.
Responding to a suspected takeover
- Contain sessions: revoke the user's IdP sessions and refresh tokens, then use the AI platform's log-out-everywhere or admin session revocation; suspend the account if activity is ongoing.
- Revoke delegated access: disconnect connectors from the user's AI account and revoke the matching OAuth grants in Google Workspace, Microsoft 365 or other platforms.
- Check for persistence: review custom instructions, memory, custom assistants, scheduled tasks, shared links, project membership and any API keys created.
- Scope exposure: pull audit logs for exports, file downloads and conversation access during the window; identify secrets, personal data and regulated data in those conversations.
- Rotate what was pasted: any credential, token or key that appears in chat history must be rotated, not just the AI account password. See the Infostealer Logs Monitoring and Response Guide.
- Clean the endpoint: if a stealer was involved, reimage the device and assume every browser-stored credential and session on it is compromised.
- Assess notification duties: personal data in chats or connected systems may trigger breach notification obligations; involve privacy and legal early.
Choosing controls by organisation size
| Situation | Minimum viable control set | Trade-off |
|---|---|---|
| Small team on a Team or Business plan | SSO with enforced domain if the plan offers it, IdP MFA with passkeys, JIT plus a monthly member review, connectors off by default | Audit logs and compliance APIs are often limited to enterprise tiers, so detection relies on IdP and endpoint signals |
| Mid-size organisation with one sanctioned assistant | Enterprise tier with SCIM, audit log export to the SIEM, connector allowlist, block consumer AI domains for corporate identities at the proxy | Blocking consumer tools without offering a sanctioned one drives use to personal devices |
| Large organisation with several AI platforms | Standard identity baseline applied to every platform, central OAuth grant inventory, session revocation automation tied to EDR and stealer log alerts | Each vendor exposes different log fields and session controls, so normalisation work is substantial |
Whatever the size, the decision that matters most is whether work use is allowed in personal accounts. If it is, none of the controls above apply to that data; if it is not, the policy needs a sanctioned alternative and discovery to enforce it. See the Shadow AI Policy Template.
Common failure patterns
- Resetting the password and stopping there: existing sessions, connector grants and modified agents survive a password change.
- SSO configured but not enforced: users keep their old email or social sign-in, and that is the credential in the stealer log.
- JIT provisioning without a leaver process: departed staff remain workspace members with full history until someone removes them.
- Assuming IdP session policy applies: the AI app's own cookie outlives the IdP session, so a replayed cookie still works.
- Ungoverned AI app OAuth grants: as in the Vercel Context.ai OAuth breach, a broad grant to an AI tool becomes the attacker's route when that vendor is breached.
- Credentials without MFA and never rotated: the Snowflake breach showed how infostealer credentials, some years old, keep working against SaaS platforms that do not enforce MFA.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev. 5 | IA-2, IA-11 Re-authentication, AC-12 Session Termination, AC-2 Account Management | SSO and MFA, session lifetime, provisioning and removal |
| NIST CSF 2.0 | PR.AA-03; DE.CM-03 | Authentication of users and services; monitoring of personnel activity |
| MITRE ATT&CK | T1539, T1550.004, T1555.003, T1528, T1078.004 | Cookie and credential theft, token theft, cloud account abuse |
| ISO/IEC 27001:2022 Annex A | A.5.18 Access rights; A.8.5 Secure authentication | Provisioning, review and removal; strong authentication |
Practitioner checklist
- Verify corporate domains on every AI platform in use and enforce SSO for them.
- Use SCIM where available; if JIT is the only option, reconcile members against the IdP weekly.
- Require phishing-resistant MFA at the IdP for AI workspace access.
- Find and migrate or close personal AI accounts on corporate email.
- Allowlist connectors, prefer read-only scopes and record each grant with an owner.
- Stream AI platform audit and authentication logs to the SIEM and alert on sign-in method drift, exports and admin changes.
- Make infostealer detection on a device trigger IdP and AI platform session revocation.
- Write an AI account takeover runbook covering sessions, connectors, agents, exports and secret rotation.
Standards and references
- Anthropic: Access audit logs
- Anthropic: Setting up JIT or SCIM provisioning
- OpenAI: Enabling multi-factor authentication
- OpenAI: Compliance Platform for Enterprise and Edu
- Microsoft Learn: Conditional Access adaptive session lifetime
- Microsoft Learn: Token Protection in Conditional Access
- OWASP: Session Management Cheat Sheet
- Group-IB: Compromised ChatGPT accounts in stealer logs
Related NHI Mgmt Group resources: LLMjacking Guide · ChatGPT Enterprise Security Guide · Claude for Enterprise Security Guide · Infostealer Logs and Session Token Abuse Guide