ChatGPT cybersecurity risks expose gaps in current IAM controls

At a glance

What this is: This is an analysis of ChatGPT-related cybersecurity risks and benefits, with the key finding that standard security controls are not enough on their own to secure AI-assisted workflows.

Why it matters: It matters because IAM, NHI, and security teams now have to govern data, access, and trust boundaries around AI tools that can sit inside ordinary business and development processes.

By the numbers:

• Debugging can absorb around 27% of developers’ time.

👉 Read Lasso Security's analysis of ChatGPT cybersecurity risks and best practices

Context

ChatGPT-related cybersecurity risk starts with a simple governance problem: once a conversational AI tool is embedded in daily work, it can move sensitive data, code, and decisions outside normal control points. That creates exposure across human identity, application access, and the broader trust boundaries that IAM and security teams are meant to enforce.

The article frames ChatGPT as both a productivity tool and a source of risk, especially when it is used to process proprietary data, generate code, or support security operations. For practitioners, the issue is not whether AI can be useful. It is whether existing identity, data handling, and review processes still hold when a user can place sensitive material into a third-party AI service in a few prompts.

Key questions

Q: How should security teams govern employee use of ChatGPT and similar AI tools?

A: Start with explicit data-handling rules, approved use cases, and logging for high-risk interactions. Identity controls tell you who used the tool, but governance must decide what they can submit, what output requires review, and which workflows are off limits. Without those boundaries, authorised use can still create leakage and unsafe decision-making.

Q: Why do standard IAM controls not fully solve AI chatbot risk?

A: Because IAM can authenticate users, but it cannot by itself control what those users paste into a model or how they interpret the response. AI risk often arises after valid access is granted, so the missing layer is policy, data classification, and workflow control around the interaction itself.

Q: What do organisations get wrong about AI-generated code?

A: They often treat it as a productivity shortcut instead of as untrusted input that can carry unsafe patterns, hidden assumptions, or manipulated logic. The safe default is to route AI-generated code through the same review and testing discipline used for any external contribution, with extra scrutiny where prompts are sensitive or untrusted.

Q: What should teams do when AI tools are used in security operations?

A: They should define where AI assistance is allowed, require verification of any recommendation before action, and restrict access to sensitive operational context. Security workflows are especially vulnerable to over-trust, so AI output must be treated as advisory until validated by a human operator.

Technical breakdown

How ChatGPT creates data exposure paths

ChatGPT processes prompts and outputs through cloud infrastructure, which means the content a user submits may traverse systems outside the organisation's direct control. The security problem is not just storage, but also transmission, logging, retention, and unintended reuse of sensitive information. When users paste personal data, source code, or commercial secrets into an AI interface, the organisation has effectively widened its data perimeter without changing its identity controls. That is why privacy policy, data classification, and access governance need to be aligned before AI use becomes routine.

Practical implication: classify what users may submit to AI tools and restrict high-sensitivity material before it enters a third-party processing path.

Why AI-generated code and malicious inputs change the threat model

The article highlights two related risks around code generation. First, an AI system can generate insecure or harmful code if the prompt or training data is compromised. Second, users can be manipulated into executing bad output when the AI appears authoritative. That shifts the threat model from simple content generation to operational influence. The relevant control question is no longer only whether code was written faster, but whether human review, secure development practices, and execution approval remain strong enough to detect unsafe output before it reaches production.

Practical implication: require human code review and test gates for AI-generated code, especially where prompts can be influenced by untrusted sources.

Why conventional controls do not fully close the AI risk gap

The article argues that antivirus, firewalls, password policy, MFA, encryption, and API security are necessary but not sufficient for ChatGPT-related risk. That is because the failure mode is not limited to external intrusion. It also includes authorized users moving sensitive data into an AI service, or trusting AI output too readily in security-sensitive workflows. This is a governance problem as much as a technical one: identity proves who accessed the tool, but not whether the resulting interaction stayed within policy.

Practical implication: add policy enforcement, monitoring, and approval rules around AI usage rather than assuming standard perimeter controls will be enough.

Threat narrative

Attacker objective: The objective is to obtain sensitive data or induce unsafe action through a trusted AI interaction path.

1. entry: The risk begins when a user submits sensitive content into a cloud-based AI service as part of an ordinary work task.
2. escalation: Manipulated prompts, unsafe output, or mishandled request state can cause confidential information to be exposed beyond the intended recipient.
3. impact: The organisation can suffer data theft, privacy violations, IP loss, or downstream security incidents when AI output is trusted or reused without review.

Breaches seen in the wild

• DeepSeek breach — DeepSeek breach exposed 1M+ log lines and sensitive secret keys.
• Cisco DevHub NHI breach — IntelBroker exploited exposed Cisco credentials, API tokens and keys in DevHub.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

ChatGPT risk is primarily a governance problem, not just a malware problem. The article shows that the main exposure comes from ordinary users placing sensitive data into an AI service and then acting on the output. That means the real control gap sits at the point where identity, data handling, and user judgement intersect. For practitioners, the lesson is that AI risk has to be governed as a business workflow, not as a narrow security add-on.

AI-assisted development expands the attack surface by accelerating both good and bad code. When developers use ChatGPT for debugging, code generation, or analysis, they also widen the scope for prompt manipulation and unsafe output. This is not a new class of risk in the abstract, but it compresses review time and increases trust in machine-produced content. The practical implication is that secure development controls must be designed for AI-mediated input, not just human-authored code.

Identity controls alone do not answer the question of acceptable AI use. MFA, encrypted transport, and API management can confirm access, but they do not determine whether a given interaction was appropriate for the data involved. That distinction matters because many AI incidents begin with authorised use. For security leaders, the governance model must extend beyond authentication into usage policy, data classification, and workflow approval.

Security teams should treat AI tools as shared enterprise services with policy boundaries, not as personal productivity apps. The article implicitly shows why uncontrolled adoption is dangerous: once many users rely on the same external AI service, the organisation inherits a common decision surface for leakage, misuse, and unsafe automation. The implication is that governance needs to scale across departments, not depend on individual judgement alone.

From our research:

• 72% of organisations have experienced or suspect they have experienced a breach of non-human identities, according to the 2024 ESG Report: Managing Non-Human Identities.
• Enterprises that have experienced a compromised NHI averaged 2.7 separate incidents in the past 12 months, according to Oasis Security & ESG.
• For the broader governance context, see Ultimate Guide to NHIs , Key Challenges and Risks for the visibility and over-privilege patterns that show why policy boundaries matter.

What this signals

AI usage will keep blurring the line between productivity and policy violation. The practical signal for programmes is that governance has to move closer to the point of interaction, especially where users can submit sensitive material into external services. The right response is to pair acceptable-use policy with monitoring and enforcement, not to assume awareness training alone will contain the problem.

ChatGPT-style workflows make policy and access control inseparable. Once a user can move internal content into a model and act on the output immediately, the organisation needs controls that cover both the input and the downstream action. That is where links to the Top 10 NHI Issues become useful, because visibility gaps and over-privilege often show up first in shared digital workflows.

Data classification has to become operational, not decorative. If teams cannot tell which content is safe to share with AI, then the tool will inherit whatever the employee decides in the moment. Align the policy to the NIST Cybersecurity Framework 2.0 and your internal review process so that AI use sits inside a clear governance boundary rather than outside it.

For practitioners

• Restrict sensitive inputs to AI tools Define which data classes users may never submit to external AI services, including personal data, regulated records, credentials, source code, and proprietary strategy documents. Back the policy with inline controls and user training so the rule is enforced before the prompt is sent.
• Require human review for AI-generated code Treat AI-generated code as untrusted until it passes the same review, testing, and approval gates as third-party code. Focus especially on code created from prompts that include untrusted context or access to internal repositories.
• Update acceptable-use controls for AI services Move AI usage into formal policy, with approved use cases, prohibited content, and escalation paths for uncertain scenarios. Make the controls visible in onboarding, security awareness, and application governance processes.
• Monitor for AI-mediated leakage and misuse Add logging and review around high-risk interactions such as uploads, long prompts containing sensitive terms, and repeated attempts to coax risky outputs. Use those signals to detect behaviour that standard perimeter tools will miss.

Key takeaways

• ChatGPT creates risk when authorised users move sensitive data, code, or decisions into an external AI service without policy boundaries.
• The article's evidence shows both productivity upside and security downside, with code work especially exposed because AI can speed output faster than review can keep up.
• Security leaders need usage policy, data classification, and review gates around AI interactions, not only perimeter controls and authentication.

Key terms

• Generative AI Risk: Generative AI risk is the possibility that a model or its users will expose data, produce unsafe output, or influence decisions in ways the organisation did not intend. In practice, the risk spans confidentiality, integrity, and governance because the model can be used correctly and still create harm through misuse or over-trust.
• Data Perimeter: A data perimeter is the boundary that determines where sensitive information may flow, be stored, or be processed. With AI tools, the perimeter matters because prompts, uploads, logs, and outputs can move information into systems outside normal enterprise control, even when the access itself is legitimate.
• AI-Mediated Workflow: An AI-mediated workflow is a business or technical process where a model contributes to a task that a human would otherwise perform directly. The governance challenge is that the model can compress review cycles and shift trust from human judgement to machine output, which changes control design.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or governance maturity in your organisation, it is worth exploring.

This post draws on content published by Lasso Security: ChatGPT & Cybersecurity: Risks, Benefits & Best Practices. Read the original.