How should organisations govern GenAI systems that interact with users in real time?

Why This Matters for Security Teams

Real-time GenAI systems do more than generate text: they interpret prompts, call tools, retrieve content, and influence downstream decisions. That means the governance problem is not limited to model quality. It includes access control, prompt handling, output review, logging, and incident response. NIST’s NIST Cybersecurity Framework 2.0 is useful here because it frames GenAI as an operational service that must be identified, protected, detected, responded to, and recovered like any other critical system.

Security teams often get caught out when the model is approved in isolation, but the surrounding workflow is not. A chatbot that reads internal documents, drafts customer replies, or triggers actions through APIs can create legal, privacy, and safety exposure even if the base model itself appears well tuned. Governance must therefore cover the whole interaction chain: user identity, allowed data, system prompts, retrieval sources, tool permissions, and escalation paths.

Practitioners also need to distinguish policy from control. A written acceptable-use statement is not enough unless the platform can enforce it through guardrails, approvals, and monitoring. In practice, many security teams encounter GenAI misuse only after an exposed workflow, harmful output, or unintended action has already occurred, rather than through intentional testing before launch.

How It Works in Practice

Effective governance starts with a clear operating model. Each GenAI use case should have a named business owner, a technical owner, a risk classification, and an approval path that reflects the sensitivity of the data and actions involved. For systems that interact with users in real time, current guidance suggests treating the prompt, retrieval layer, tool layer, and response layer as separate control points rather than one combined feature.

NIST’s NIST AI 600-1 GenAI Profile is particularly useful because it translates AI risk management into practical governance activities. That includes pre-deployment testing for prompt injection, jailbreaks, unsafe completions, and data leakage, followed by continuous monitoring for drift in behaviour, policy bypasses, and changes in retrieval quality. Security teams should also define when human review is mandatory, especially for regulated advice, external communications, and any action that changes records or entitlements.

At the control level, organisations should align the GenAI stack to established security disciplines rather than inventing a separate programme from scratch. That usually means:

• restricting who can configure prompts, tools, and knowledge sources
• logging prompts, outputs, tool calls, and override actions for investigation
• rate limiting and abuse detection for high-volume or automated interaction patterns
• red teaming against realistic attacker goals, not only generic toxicity tests
• placing approval gates around external side effects such as sending messages or updating tickets

For deeper control mapping, NIST SP 800-53 Rev 5 Security and Privacy Controls remains the most practical anchor for access, logging, configuration management, incident response, and system integrity. These controls tend to break down when real-time GenAI is embedded in legacy business applications because the model layer, API layer, and workflow layer are often owned by different teams with no shared control boundary.

Common Variations and Edge Cases

Tighter governance often increases latency, review overhead, and product friction, requiring organisations to balance user experience against risk containment. That tradeoff becomes sharper in customer-facing or employee-assist scenarios where response time matters and business teams want broad autonomy. Best practice is evolving, but there is no universal standard for how much autonomy a live GenAI system should have before it needs human intervention.

Edge cases usually appear where the system is connected to sensitive data, high-impact decisions, or privileged actions. A support assistant that only summarises public content has a very different risk profile from one that can access internal files, create refunds, or modify cloud resources. If user identity and authorisation are weak, the GenAI layer can become a shortcut around existing controls rather than an extension of them.

Another common blind spot is vendor-managed or embedded GenAI. Organisations may assume the provider has handled safety, but accountability still sits with the deploying organisation for use-case selection, data handling, and user impact. Where the system supports regulated activity, the governance model should also define retention, auditability, appeal routes, and fallback procedures when the model is unavailable or confidence is low. In practice, those gaps show up first in exception handling, not in the happy-path demo.

Related resources from NHI Mgmt Group

• How should security teams govern systems where business rules change in real time?
• How should organisations govern AI systems that learn environment state over time?
• How should organisations handle identity verification when deepfakes can mimic real users?
• How should organisations govern GenAI before broad rollout?