Who is accountable when an AI agent in ChatGPT Enterprise exposes sensitive information?

Why This Matters for Security Teams

Accountability is the deciding factor when a ChatGPT Enterprise agent exposes sensitive information because the incident usually reflects an approval, design, or monitoring failure rather than a single user mistake. Autonomous tools can surface data through prompt chaining, connector misuse, or overly broad workspace permissions, which means governance cannot stop at “who clicked send.” The risk profile is similar to the control failures described in the State of Secrets in AppSec, where secret leakage often persists longer than teams expect.

Current guidance from the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework points toward shared accountability across security, IAM, governance, and the business owner that requested the workflow. In practice, the issue is not whether the model is “smart,” but whether the organisation defined boundaries tightly enough to prevent accidental disclosure through tools, memory, or data connectors. In practice, many security teams encounter this only after a sensitive prompt, file, or connector has already exposed data beyond its intended audience.

How It Works in Practice

For enterprise AI agents, accountability should map to the control layer that allowed the exposure. The business owner defines the use case and acceptable data scope. Security and IAM teams design the access model, including connector permissions, workspace segregation, and logging. Governance teams approve the policy, retention rules, and review cadence. If the agent was allowed to access sensitive content without a clear business need, the accountability failure is usually organisational, not technical.

Practically, teams should treat the agent as a governed workload rather than a chatbot. That means assigning a named owner, recording the approved data classes, and requiring policy checks before connectors can retrieve files, messages, tickets, or knowledge-base content. It also means logging the agent’s tool calls, not just the user prompt, because exposure often happens during retrieval and post-processing. The CSA MAESTRO agentic AI threat modeling framework and MITRE ATLAS adversarial AI threat matrix both reinforce that agent behavior must be assessed across tools, memory, and identity boundaries.

• Require a human owner for each enterprise agent workflow.
• Scope access to the minimum data set needed for the task.
• Review connector permissions before deployment and after material workflow changes.
• Monitor prompts, tool calls, and outputs for sensitive-data exposure.
• Revoke or redesign access when the agent’s behaviour expands beyond the approved use case.

NHIMG research also shows why speed matters: the LLMjacking: How Attackers Hijack AI Using Compromised NHIs analysis documents how quickly exposed credentials can be abused once they are reachable. These controls tend to break down when enterprise agents are connected to broad document stores, email, or ticketing systems because retrieval pathways expose far more information than the original prompt suggests.

Common Variations and Edge Cases

Tighter AI governance often increases operational overhead, requiring organisations to balance rapid adoption against review, logging, and access-friction. That tradeoff becomes especially sharp when a ChatGPT Enterprise deployment is embedded in customer support, engineering, or legal workflows, where staff expect the agent to “just know” more than it should. Best practice is evolving, and there is no universal standard for assigning liability across every enterprise AI scenario yet.

Edge cases usually involve shared workspaces, delegated administration, or mixed human-and-agent workflows. If a user grants access to a file and the agent later republishes the information in a summary, the root cause may be weak data classification, not malicious intent. If an administrator approves a connector that can reach regulated content, the governance gap is different again. Organisations should also distinguish between policy accountability and legal accountability: the business may own the use case, while security owns the safeguards and auditability. The OWASP NHI Top 10 is useful here because it frames agentic exposure as an identity and authorization problem, not just a model-quality issue.

In practice, the best answer is to preassign accountability before deployment, then revisit it whenever the agent gains a new connector, a broader data scope, or autonomous tool execution.

Related resources from NHI Mgmt Group

• Why is single-provider AI agent governance not enough for enterprise security?
• Who is accountable when AI search exposes sensitive enterprise data?
• Who is accountable when AI agent access to SaaS data exposes regulated information during audit scope?
• Who is accountable when an AI agent exposes sensitive Supabase data through MCP access?