A common mistake is assuming the model layer is the main security issue. In practice, the bigger problems are identity sprawl, overbroad permissions, data leakage, and unclear human accountability. Teams should validate who can prompt, what systems the AI can reach, which records it can process, and how exceptions are approved.
Why This Matters for Security Teams
Government GenAI deployments often fail for the same reason many identity programs fail: teams focus on the model and ignore the operating context around it. The real risk is not just prompt quality, but which users can invoke the system, which records it can read, and which downstream services it can touch. NIST’s Cybersecurity Framework 2.0 reinforces that governance, access control, and monitoring must be built into the workflow, not bolted on afterward.
That matters because public-sector workflows often span case files, citizen records, approvals, and interagency data exchanges. A GenAI tool with overbroad access can surface information across boundaries that were never meant to be crossed by a single user request. NHIMG’s Top 10 NHI Issues repeatedly shows that identity sprawl and weak lifecycle controls are the common failure mode, not model novelty. In practice, many security teams discover the problem only after a workflow has already exposed sensitive data or bypassed a human approval step.
How It Works in Practice
Safe deployment starts with treating the GenAI application as a governed workload, not a generic chatbot. The system should have a defined identity, narrow runtime permissions, and explicit approval paths for sensitive actions. That means separating who can prompt the system from what the system can retrieve, generate, or execute. The model may be probabilistic, but access decisions should not be. Current guidance from the NIST AI 600-1 GenAI Profile and NIST SP 800-53 Rev. 5 is to pair AI governance with standard access, audit, and data protection controls.
In operational terms, that usually means:
- Binding each workflow to a distinct non-human identity rather than a shared service account.
- Using just-in-time access for sensitive records and revoking access when the task ends.
- Classifying which datasets the model may process, especially records containing personal, legal, or classified information.
- Logging prompts, tool calls, retrievals, and human approvals so investigators can reconstruct the decision path.
- Putting exception handling behind human review, especially when the system requests escalation or cross-domain access.
NHIMG’s Ultimate Guide to NHIs — Lifecycle Processes for Managing NHIs is useful here because GenAI workflows inherit the same lifecycle problems as other machine identities: issuance, rotation, revocation, and ownership. The lesson from the DeepSeek breach is that exposed secrets and uncontrolled data paths quickly turn an AI system into an enterprise-wide exposure point. These controls tend to break down when multiple agencies share one workflow account because accountability, logging, and privilege boundaries all collapse together.
Common Variations and Edge Cases
Tighter control often increases deployment friction, requiring organisations to balance speed of adoption against auditability and access discipline. That tradeoff is real in government, where a single workflow may need to serve public contact centres, back-office case handling, and restricted internal review. Best practice is evolving, but there is no universal standard for exactly where to place the boundary between automation and human approval.
The hardest edge cases are cross-domain workflows and exception-heavy processes. A citizen-service assistant may need access to one dataset for triage, then a different approval chain for redaction, then a separate system for case closure. If the same identity can traverse all three steps, privilege accumulates silently. The Ultimate Guide to NHIs — Regulatory and Audit Perspectives and the Indian Government Breach both point to the same operational lesson: auditability and ownership matter as much as technical capability. Teams also need to plan for vendor-hosted models, where logging, residency, and retention settings may not match public-sector obligations. Where data classification is incomplete or exception handling is informal, the guidance breaks down because the system cannot reliably know what it is allowed to see or do.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, CSA MAESTRO and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | A01 | GenAI workflows fail when agent actions are not tightly constrained. |
| CSA MAESTRO | M4 | Governance must cover agent identity, access, and execution boundaries. |
| NIST AI RMF | AI risk management addresses accountability, monitoring, and misuse in GenAI deployments. | |
| OWASP Non-Human Identity Top 10 | NHI-01 | Shared service accounts and leaked secrets are core NHI deployment risks. |
| NIST CSF 2.0 | PR.AC-4 | Least-privilege access is central to limiting GenAI workflow exposure. |
Map governance, measurement, and oversight controls before expanding GenAI into public workflows.
Related resources from NHI Mgmt Group
- What do organisations get wrong about adding AI to existing workflows?
- What do organisations get wrong about shared accounts in high-friction workflows?
- What do organisations get wrong about biometric verification in remote workflows?
- What do organisations get wrong about enforcing security in developer workflows?
