Compliance review checks whether a deployment was approved and documented. Runtime control checks whether the chatbot was actually constrained during use, including data masking, access scope, and output filtering. In regulated healthcare, runtime control matters more because the risk occurs during the interaction, not just at launch.
Why This Matters for Security Teams
Chatbot compliance review answers a governance question: was the system approved, documented, and placed under the right oversight? Runtime control answers a security question: what was the chatbot allowed to do at the moment it handled a prompt, tool call, or data lookup? That distinction matters because regulated workflows fail in execution, not paper trails. A chatbot can pass review and still expose PHI, overreach its access scope, or produce unsafe output if controls are not enforced live. Current guidance in NIST Cybersecurity Framework 2.0 and Ultimate Guide to NHIs — Regulatory and Audit Perspectives points to evidence of operating control, not just approval evidence.
For AI chatbots, especially those connected to internal records, runtime control is the practical test of least privilege, data minimisation, and output governance. Compliance review can show intent and due diligence, but it does not prove masking rules were active, secrets were scoped correctly, or filters held under pressure. In practice, many security teams encounter the gap only after a chatbot has already disclosed something it should never have been able to access.
How It Works in Practice
In implementation terms, compliance review is mostly static: policy approval, architecture review, risk sign-off, and evidence collection. Runtime control is dynamic: it enforces policy every time the chatbot receives input, calls a tool, or returns output. That means the control plane must sit in front of model use, retrieval paths, and downstream systems so policy decisions happen in real time, not only in design documents.
For regulated healthcare, practical runtime control usually includes four layers:
- Input controls: mask or classify sensitive data before it reaches the model.
- Access controls: restrict retrieval, tool use, and API scope to what the session needs.
- Output controls: block PHI leakage, unsafe advice, and disallowed content before delivery.
- Session controls: short-lived credentials, logged decisions, and revocation when the task ends.
This is where NHI thinking becomes useful. A chatbot or agent should not be treated as a trusted application with broad standing access. Ultimate Guide to NHIs — What are Non-Human Identities and Ultimate Guide to NHIs — Lifecycle Processes for Managing NHIs emphasise that non-human access must be governed across issuance, use, rotation, and revocation. In practice, that means JIT credentials, workload identity, and policy checks at request time, rather than a one-time approval. For agentic systems, current guidance suggests aligning this with NIST Cybersecurity Framework 2.0 and agent security work such as OWASP and CSA MAESTRO, which both favour continuous verification over static trust.
That model works best when the chatbot has clear, narrow tasks and deterministic tool paths. These controls tend to break down when the chatbot can chain prompts, call multiple tools, or reach nested systems because the effective access path changes faster than a pre-approved review can capture.
Common Variations and Edge Cases
Tighter runtime control often increases operational overhead, requiring organisations to balance safety against latency, usability, and support burden. That tradeoff becomes visible when a chatbot serves clinicians, claims staff, or patient-facing workflows where even a small delay or false block can interrupt care. Best practice is evolving, but there is no universal standard yet for exactly how much filtering or how much context-aware authorisation is enough.
One common edge case is “review-only” governance for low-risk internal bots. That can be acceptable for non-sensitive content, but it should not be confused with control over PHI, secrets, or privileged workflows. Another edge case is vendor-hosted chat interfaces: a compliance packet may look strong, while the real question is whether the deployed session can be constrained at runtime. The Top 10 NHI Issues resource and the industry’s breach data show why static approval is not enough when secrets, scopes, and tool access drift over time. In those environments, runtime controls should be tested with live prompts, live roles, and live data paths, not only with policy documents and screenshots.
Where the chatbot behaves more like an autonomous agent than a fixed Q&A widget, guidance from Ultimate Guide to NHIs — Standards points toward continuous enforcement, not one-time approval.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and CSA MAESTRO address the attack and risk surface, while NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | Runtime policy enforcement is central to agentic chatbot safety. | |
| CSA MAESTRO | Covers agent governance, tool use, and runtime containment. | |
| NIST AI RMF | Supports governance and risk controls for AI systems in operation. |
Track operational AI risks continuously and require evidence of live control effectiveness.
Related resources from NHI Mgmt Group
- What is the difference between compliance evidence and runtime access control?
- What is the difference between attack surface management and NHI governance?
- What is the difference between reviewing human access and reviewing NHIs?
- What is the difference between role-based access and API key governance for NHI security?
