Use layered controls, not a single moderation rule. Define policy boundaries, test them with red teaming, monitor model behaviour in production, and assign ownership for exceptions and escalations. Safe deployment depends on proving that the model can resist manipulation, not just that it can filter obvious abuse.
Why This Matters for Security Teams
Public-facing generative ai changes the risk profile from a static application to a dynamic service that can be manipulated through prompts, context, and downstream integrations. A single policy filter is rarely enough because the exposure includes prompt injection, unsafe tool use, data leakage, and misleading outputs that can affect customers, staff, and legal obligations. NIST’s NIST Cybersecurity Framework 2.0 is useful here because it treats governance, identification, protection, detection, response, and recovery as linked functions rather than isolated tasks.
The practical problem is that teams often treat the model as the control boundary, when the real boundary includes prompts, retrieval sources, plugins, action tools, approval paths, and human escalation. That creates a governance gap if ownership is unclear or if safety testing stops after launch. Organisations also need to decide which harms matter most: hallucinated advice, abuse of brand trust, exposure of sensitive data, or unauthorised actions triggered by an agentic workflow. In practice, many security teams encounter these failures only after a public output or tool action has already created customer impact, rather than through intentional pre-production abuse testing.
How It Works in Practice
Safe governance starts by defining the system boundary. Teams should document what the model is allowed to say, what it must refuse, what data it can see, and which tools it can invoke. That policy should be enforced at multiple layers: input filtering, retrieval filtering, output checking, rate limits, human review for sensitive actions, and logging for investigation. For GenAI-specific governance, NIST’s NIST AI 600-1 Generative AI Profile and NIST AI 600-1 GenAI Profile both reinforce the need to manage risks across the AI lifecycle, not only during prompt design.
A workable operating model usually includes the following controls:
- Policy design that separates acceptable use, prohibited use, and escalation-only requests.
- Red teaming that tests jailbreaks, prompt injection, data extraction, and unsafe tool invocation.
- Content and action monitoring that looks for abuse patterns, not just offensive words.
- Exception handling with named owners for overrides, model changes, and incident review.
- Change management for prompts, retrieval corpora, guardrails, and model versions.
Security teams should also treat logs as evidence, not just telemetry. If a public model can retrieve internal knowledge, generate regulated advice, or trigger workflows, then the organisation needs a reviewable record of who changed what and why. That maps well to control discipline in NIST SP 800-53 Rev 5 Security and Privacy Controls, especially where auditability, access control, and system integrity matter. These controls tend to break down in fast-moving product teams that deploy multiple model variants and toolchains without a single approval path because no one can reliably trace which prompt, retrieval source, or action policy produced the unsafe result.
Common Variations and Edge Cases
Tighter governance often increases latency, false refusals, and operational overhead, requiring organisations to balance user experience against safety and accountability. That tradeoff is especially visible in customer-facing assistants, where aggressive filtering can frustrate legitimate users while loose controls can expose the business to abuse.
Current guidance suggests there is no universal standard for how much human review is enough. High-risk use cases, such as financial guidance, healthcare triage, or workflows that can trigger external actions, usually need stronger approval gates and stricter monitoring than general-purpose chat. Lower-risk informational assistants may rely more heavily on post-generation checks and rate-limited access, but only if the organisation has defined a clear tolerance for error.
Edge cases also matter. Retrieval-augmented systems can look safe in testing yet fail when the underlying knowledge base contains stale, untrusted, or adversarial content. Agentic systems raise the bar further because the security question is not only what the model says, but what it can do through connected tools. Where public-facing AI exposes regulated data, personal data, or privileged business systems, identity and access governance becomes part of the AI control set, not a separate concern. The safest programmes keep policy, identity, and monitoring aligned from the start, rather than bolting them on after launch.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 address the attack and risk surface, while NIST AI RMF, NIST AI 600-1, NIST CSF 2.0 and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI risk governance is central to public-facing GenAI safety. | |
| NIST AI 600-1 | The GenAI profile addresses operational controls for generative systems. | |
| NIST CSF 2.0 | GV.OC, DE.CM, RS.MI | Governance, monitoring, and response map directly to public AI operating controls. |
| NIST SP 800-53 Rev 5 | AU-2, AC-6, SI-4 | Logging, least privilege, and monitoring support safe AI deployment. |
| OWASP Agentic AI Top 10 | Agentic AI threats cover prompt injection and unsafe tool use. |
Test agent workflows for injection, tool abuse, and unintended side effects before release.
