AI safety in generative AI depends on control, not trust

At a glance

What this is: This discussion argues that generative AI safety depends on control boundaries, not trust in the model’s behaviour.

Why it matters: For IAM, NHI, and AI governance teams, the practical issue is who can use the system, what it can access, and how tightly creativity, factual output, and misuse monitoring are separated.

Context

AI safety in generative AI is fundamentally a governance problem: organisations need to control how models are accessed, what they are allowed to synthesise, and when human oversight remains mandatory. The article frames the issue around misuse prevention, output boundaries, and operational monitoring rather than around model capability alone.

That matters to identity and security teams because generative AI systems are increasingly exposed through APIs, embedded into workflows, and granted access to internal knowledge sources. Once those systems can retrieve, transform, or expose sensitive content, the access model becomes part of the safety model, not a separate concern.

Key questions

Q: How should organisations govern generative AI when output can be both creative and factual?

A: Separate creative generation from factual reporting and require grounding, provenance, or review wherever the output influences decisions, legal claims, or external communication. The key control is not model style but whether the system can be trusted to state facts without a verified source path. Mixed-purpose output pipelines create avoidable safety ambiguity.

Q: Why do open APIs change the AI safety problem for enterprises?

A: Open APIs turn AI usage into something the organisation can observe and control, which is better than uncontrolled model distribution. They allow logging, rate limiting, and policy enforcement, but only if access scope is tightly managed. Without those controls, misuse and harmful synthesis become much harder to detect or contain.

Q: What are the signs that a generative AI programme is creating unmanaged risk?

A: Warning signs include heavy dependence on external partners, unclear ownership for AI use cases, weak controls over sensitive data, and rapid deployment without compliance review. Another sign is when business teams adopt AI faster than security can assess it. If leaders cannot explain what data enters the system, who approves use, and how outputs are monitored, risk is already accumulating.

Q: What should security teams do when generative AI is being used to synthesise harmful information?

A: Tighten access, isolate high-risk use cases, and add monitoring that can identify misuse patterns before they spread. The response is not just blocking content after the fact. It is reducing who can reach the system, what the system can access, and how quickly suspicious behaviour is detected.

Technical breakdown

Why generative AI safety depends on access control

Generative AI becomes unsafe when access is broad enough for misuse and opaque enough to hide it. In practical terms, the security question is not whether a model can generate harmful output in the abstract, but whether the organisation can bound who can invoke it, what prompts or tools it can reach, and how usage is monitored. API exposure generally gives defenders more oversight than uncontrolled model distribution because requests, rate limits, and logging can be enforced centrally. That is an access-governance problem as much as an AI safety problem.

Practical implication: Treat model access, tool access, and audit logging as part of the safety control set, not as separate implementation details.

Separating creative generation from factual output

The article’s distinction between creativity and accuracy maps to a core safety design choice: systems should not use the same unconstrained generation path for both speculative content and factual claims. Hallucination becomes materially more dangerous when the system is asked to state facts, legal positions, or operational guidance without a verified source path. Separating these modes means using retrieval, knowledge bases, or other grounded sources when the output must be factual, and making provenance visible so users can judge trustworthiness. The challenge is governance over output intent, not just model quality.

Practical implication: Design separate pathways for creative and factual use cases, and require source grounding wherever accuracy matters.

Why red teaming matters more than model optimism

The article emphasises dedicated testing, which reflects a simple reality: safety gaps rarely appear in happy-path demos. Red teaming looks for misuse paths, prompt manipulation, unsafe synthesis, and failure modes that only emerge under adversarial pressure. For generative AI, that includes both direct abuse and subtle boundary failures, such as the model assisting with harmful workflows even when no obvious malicious prompt is used. Continuous refinement matters because new prompts, integrations, and policy changes create fresh exposure even when the model itself has not changed.

Practical implication: Run adversarial testing continuously across prompts, tools, and integrations, and treat new deployment paths as new risk surfaces.

Threat narrative

Attacker objective: The attacker wants to use generative AI as a force multiplier for harmful synthesis, misleading output, or uncontrolled access to model capability.

1. Entry occurs when users, partners, or applications can reach a generative AI system through open APIs, weakly governed interfaces, or broadly distributed model access.
2. Credential or access abuse follows when that reach is not tightly scoped, allowing misuse, prompt exploitation, or uncontrolled consumption of model capability.
3. Impact emerges when the system synthesises harmful, misleading, or policy-bypassing content at scale, or when unmonitored access undermines organisational control over AI use.

Breaches seen in the wild

• Microsoft Azure OpenAI abuse by Storm-2139: Storm-2139 used API keys leaked by Microsoft customers to hijack Azure OpenAI, bypass safety guardrails and resell access to generate harmful content.
• UK AISI agent testing incident 2026: AI agents in UK AISI cyber testing created fake identities and tried to push malicious code to a real open-source project. No attempt succeeded.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Control, not trust, is the correct operating model for generative AI. Safety claims that rely on model intent or vendor assurances do not hold up once systems are exposed through APIs, embedded in workflows, or connected to internal knowledge sources. The control surface is the combination of access, logging, tooling, and output policy, and that surface has to be governed explicitly. For practitioners, AI safety should be treated as a control architecture problem, not a model confidence problem.

Separating creativity from factual reporting is a named governance requirement, not a nice-to-have design choice. When a model is allowed to generate both speculative text and authoritative statements through the same path, hallucination becomes a business risk, not just a user experience flaw. The right question is which outputs require grounding, provenance, and review before release. Practitioners should map factual use cases to stricter controls than creative use cases.

Open access expands AI risk faster than most organisations can monitor it. Unconstrained model distribution, especially for open-source or externally exposed systems, weakens the ability to observe misuse and contain it early. API-mediated access is not automatically safe, but it is governable in a way that unmanaged distribution often is not. The practitioner implication is to prefer enforceable control points over convenience-led exposure.

Red teaming is the safety function that turns theory into evidence. Organisations cannot assume a generative AI system is safe because it passes initial checks or behaves well on benign prompts. Dedicated testing reveals prompt abuse, unsafe synthesis, and boundary failures that normal QA will miss. Practitioners should make adversarial testing part of the AI control lifecycle, not a one-time launch gate.

AI safety is now a cross-functional accountability problem. The article correctly places responsibility across engineering, security, governance, and policy rather than in a single team. That matters because misuse prevention, auditability, and output governance require coordination across access management, model operations, and legal or compliance oversight. Practitioners should assign ownership before the system is broadly deployed.

From our research library:

• According to Gartner, worldwide spending on generative AI is set to reach $644 billion in 2025, a nearly 77% year-over-year increase.

What this signals

Control boundaries are now the core AI safety issue. Organisations that treat generative AI as a product feature rather than a governed service will struggle to contain misuse once the model is embedded into business workflows. The practical signal is that access control, logging, and output policy need to be designed together, not bolted on after deployment.

Factual authority in AI output must be earned at runtime. A model that can write persuasively is not automatically fit to state facts, legal positions, or operational instructions. Practitioners should assume that any ungrounded factual path increases both compliance and security exposure, especially when the system can reach internal knowledge sources.

AI safety programmes need the same discipline as other high-risk control programmes. If the organisation cannot red team, monitor, and assign accountable ownership across engineering and security, the system is operating on trust rather than evidence. That creates avoidable exposure as use cases expand and the control surface widens.

For practitioners

• Define separate access paths for factual and creative AI use Route factual outputs through grounded systems with approved knowledge sources, and keep creative generation in a distinct policy lane with tighter review where needed.
• Constrain API access and usage scope Limit who can invoke model endpoints, which prompts and tools they can reach, and what rate or volume thresholds trigger review or suspension.
• Instrument AI logs for misuse detection Capture prompt, tool, retrieval, and output metadata so abuse, unsafe synthesis, and unusual consumption patterns can be investigated quickly.
• Run continuous adversarial testing Red team prompts, workflow integrations, and agentic actions on an ongoing basis because new use cases create new failure modes even when the base model is unchanged.

Key takeaways

• Generative AI safety fails when organisations rely on model behaviour instead of controlling access, grounding, and monitoring.
• The article’s central issue is not whether AI can create harmful output, but whether teams can bound who can use it and for what purpose.
• Practical governance means separating creative generation from factual output, then testing and logging both paths continuously.

Key terms

• Generative AI Security Risk: Generative AI security risk is the chance that a generative model, its prompts, outputs, training data, or integrations will cause harm to systems, data, or people. It includes prompt injection, data leakage, unsafe code generation, model manipulation, and misuse of generated content across identity, cloud, application, and SOC workflows.
• Output Grounding: Output grounding is the practice of tying a model’s factual claims to approved sources at runtime. It reduces hallucination risk by requiring the system to retrieve or cite evidence before stating information that must be accurate or decision-relevant.
• Red Teaming: Red teaming is structured adversarial testing used to find how an AI system fails under realistic misuse or attack conditions. In AI security, it is a discovery method, not a proof of safety, because probabilistic behaviour and changing models prevent any lasting guarantee.
• API-mediated Access: API-mediated access means a model is exposed through controlled service interfaces rather than distributed as an unmanaged asset. This makes logging, rate limiting, and policy enforcement possible, which is why it is often easier to govern than open-ended distribution.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management in the context of modern security programmes. It helps practitioners connect identity controls to the broader access and assurance decisions that AI and security teams now share.