Chatgpt trust depends on verification, not blind confidence

At a glance

What this is: This article argues that trust in ChatGPT should be established through independent verification rather than assumptions about model accuracy.

Why it matters: It matters to IAM practitioners because AI use intersects with identity trust, data sharing, access decisions, and the security posture of both human and non-human identity programmes.

By the numbers:

• 90% of IT leaders say properly managing NHIs is essential for a successful zero-trust implementation.
• 79% of organisations have experienced secrets leaks, with 77% of these incidents resulting in tangible damage.
• Only 5.7% of organisations have full visibility into their service accounts.

👉 Read DigiCert's blog post on establishing identity and security while using AI

Context

ChatGPT and similar tools are useful because they compress research, summarisation, and drafting into a single interface, but that convenience does not create trust. The security problem is that generated answers can be plausible, incomplete, outdated, or simply wrong, which means the output itself becomes an unverified input to business decisions.

For IAM teams, that creates a familiar problem in a new form: identity and trust controls still need to determine what can be relied on, who can validate it, and what data may be shared with the system. The article's core point is that AI output should be treated like any other externally sourced assertion until it has been independently checked.

This is a human identity article with direct implications for NHI and autonomous programmes because the same trust discipline applies across users, service accounts, and AI-assisted workflows. As with access governance, the issue is not whether the system is useful, but whether the organisation can prove what it should trust and why.

Key questions

Q: How should organisations govern employee use of ChatGPT for security work?

A: Treat ChatGPT as an untrusted drafting and research aid, not as an authority. Limit the kinds of tasks it can support, ban the sharing of secrets or sensitive operational data, and require verification against trusted sources before any output is used in access, incident, or policy decisions.

Q: Why do AI chat tools create risk for identity and access teams?

A: They create risk because users may rely on plausible but unverified output when making identity, access, or security decisions. That can lead to bad approvals, weak guidance, or sensitive data disclosure. The control problem is trust discipline, not just model quality.

Q: How can security teams tell whether AI output is trustworthy enough to use?

A: They should look for an external verification path. The answer should be grounded in a trusted source, a documented control, or a human review step. If the output cannot be validated independently, it should be treated as advisory only and not operationalised.

Q: What should employees avoid sharing with ChatGPT and similar tools?

A: Employees should avoid sharing secrets, credentials, internal code, personal data, and confidential business information unless the organisation has approved that specific platform and use case. Anything that would be unsafe to post publicly should be considered unsafe to prompt into an AI tool.

Technical breakdown

Why AI outputs need independent verification

Large language models generate responses by predicting likely text from training patterns, not by asserting verified facts. That means they can sound confident while being wrong, stale, or incomplete. In security and identity workflows, this is dangerous because a fluent answer can be mistaken for an authoritative one. Independent verification is the control that separates usable assistance from untrusted content. In practice, the verification source may be a certificate authority, an authoritative data source, or a human approval path, depending on the use case.

Practical implication: require a second trust signal before any AI-generated output is used to make identity, access, or security decisions.

How AI creates new trust and privacy exposure

The article highlights a second issue beyond answer quality: data disclosure. When users share sensitive details with a chat system, that information may be retained, reused, or exposed in future interactions depending on the platform's handling model. In security terms, the problem is not only output integrity but input discipline. If a user would not place information into a public ticket or external forum, they should not treat an AI prompt as a safe exception.

Practical implication: define prompt-handling rules for sensitive identity, secrets, and operational data before AI becomes embedded in daily work.

Where digital trust fits into AI governance

The article makes a PKI argument: if trust is the issue, verification mechanisms matter. Digital certificates already prove domain ownership, legal identity, and other attributes in machine-readable form. That pattern can be extended to AI-related assertions, provided the organisation knows which statements must be verified and who owns that verification. This is less about trusting the model and more about trusting the assertion path around the model.

Practical implication: map high-risk AI use cases to explicit verification controls, including certificate-based trust where machine validation is feasible.

NHI Mgmt Group analysis

Trust in AI output is an identity problem before it is a model problem. The article correctly shifts attention away from whether ChatGPT is clever and toward whether its outputs can be trusted in operational use. That is the same governance question IAM teams ask for credentials, certificates, and service accounts: what assertion is being accepted, and by whom. The practitioner conclusion is that AI output without verification should never be treated as a trusted control input.

Human trust assumptions break first when AI becomes part of everyday work. People are conditioned to treat fluent language as knowledgeable language, which makes AI-generated content especially risky in security, legal, and access-adjacent workflows. This is a human IAM problem because the failure mode sits in user judgement, not only in tooling. The practitioner conclusion is that organisations need clear prompt hygiene, review expectations, and escalation paths for AI-assisted decisions.

Independent verification is the durable trust pattern, not model confidence. The article's PKI framing is strong because it re-centres trust on cryptographic and organisational proof rather than probabilistic output quality. In identity programmes, that means verification must remain external to the thing being verified. The practitioner conclusion is that AI governance should reuse established trust primitives wherever possible.

ChatGPT-style risk is a preview of broader non-human trust debt. Once organisations accept unverified machine output in one workflow, the same tolerance spreads to automated assistants, workload identities, and agentic systems that act on it. The bigger issue is not one chatbot, but the growing habit of treating machine-generated assertions as self-authenticating. The practitioner conclusion is to align AI governance, NHI governance, and human review around the same verification standard.

Verification must become a policy control, not an ad hoc habit. The article shows that trust cannot be left to individual users deciding when a tool feels reliable enough. That approach creates uneven risk across teams and use cases. The practitioner conclusion is to define which AI outputs require source checking, which require human approval, and which are prohibited from containing sensitive data.

From our research:

• 90% of IT leaders say properly managing NHIs is essential for a successful zero-trust implementation, according to Ultimate Guide to NHIs.
• 71% of NHIs are not rotated within recommended time frames, increasing the risk of compromise over time, according to our research.
• For a broader control baseline, Ultimate Guide to NHIs , What are Non-Human Identities explains the identity types that zero-trust programmes must cover.

What this signals

Verification debt: organisations that normalise AI-generated answers without source checking accumulate a trust gap that looks different from classic IAM drift but behaves the same way at the decision layer. The practical response is to codify which outputs need proof, not just which systems need access.

The identity programme should treat AI output as an untrusted assertion unless it is tied to a verifiable source. That discipline aligns with the NIST Cybersecurity Framework 2.0 NIST Cybersecurity Framework 2.0 functions for governance and protect, because trust is being extended to information, not just accounts.

If AI is being used anywhere near credentials, access reviews, or policy drafting, the real control question is whether a human or machine can independently validate the statement before it influences action. That is where human identity practice, NHI governance, and emerging agentic workflows start to converge.

For practitioners

• Define approved AI use cases Classify which identity, security, and operations tasks may use chat-based assistance and which require authoritative sources or human review before action is taken.
• Set prompt data-handling rules Prohibit the entry of secrets, credentials, personal data, and internal operational details unless the platform has been formally approved for that data class.
• Add verification steps to AI-assisted workflows Require users to confirm model outputs against a trusted source before they are used in tickets, approvals, change records, or access decisions.
• Teach staff how AI-assisted phishing works Update awareness training so employees can recognise AI-written scams, fabricated citations, and messages that imitate internal or family voices.

Key takeaways

• AI tools become a security problem when their outputs are trusted without independent verification.
• The scale of the risk is practical, not theoretical, because users may share sensitive data or act on false output.
• Identity and security teams should codify prompt rules, source checks, and human review before AI output influences action.

Key terms

• Independent Verification: A trust pattern in which an assertion is checked against a separate source before it is accepted. In identity and security programmes, this prevents fluent but unproven information from being treated as authoritative. It applies to AI outputs, certificates, and other machine-generated claims.
• Digital Trust: The confidence that a digital assertion, transaction, or identity claim is authentic and reliable. In practice, digital trust comes from cryptographic proof, authoritative validation, and governance controls rather than from the apparent confidence of the system making the claim.
• Prompt Hygiene: The discipline of deciding what can and cannot be shared with an AI tool during prompting. It covers sensitive data exclusion, approved use cases, and review expectations so that employees do not turn chat interfaces into uncontrolled data sinks.
• Verification Path: The specific route by which an AI-generated statement is checked before it influences a decision. A strong verification path uses a trusted source, a documented process, or a human control rather than relying on model confidence or convenience.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or governance in your organisation, it is worth exploring.

This post draws on content published by DigiCert: How Much Can You Trust ChatGPT? Establishing identity and security while using AI. Read the original.