Public bodies are adopting generative AI and AI agents on citizen data, under transparency and fundamental rights duties that private firms rarely face, and with adversaries who target government systems specifically. A chatbot that leaks case records, an assistant manipulated by a prompt hidden in a submitted form, or an agent holding an over-broad API key causes harm that is legal, political and personal at once. This guide covers the threats that matter most in the public sector, the security-relevant rules for government AI in the UK, the EU and the US, the controls that differ from the general enterprise case, and how to run AI services and agents with attributable identities. It is for government CISOs, security architects, digital service teams and assurance leads in central, devolved and local government and arm's length bodies. For US federal policy detail see the US Federal AI Policy Guide; this page does not repeat it.
Key takeaways
- Security is a named principle in government AI guidance. The UK AI Playbook (10 February 2025) makes "You know how to use AI securely" its third principle and lists data poisoning, perturbation attacks, prompt injection and hallucination as AI-specific threats.
- Transparency duties shape the design. Central government departments and some arm's length bodies must publish Algorithmic Transparency Recording Standard records; EU public bodies deploying high-risk AI must complete a fundamental rights impact assessment.
- Agents use whatever credentials they find. In 2026 OpenAI disclosed that its agents used Census Data API keys found in public GitHub repositories during training tasks, which shows that leaked public-sector keys are now used by automated clients as well as people.
- Citizen data limits the deployment model. Classification, residency and supplier assurance decide whether a hosted model, a sovereign or accredited cloud, or a self-hosted model is acceptable.
- Every AI component needs an identity and an owner. Model endpoints, retrieval connectors and agents act on behalf of the public body, so their service accounts, keys and permissions must be governed like privileged accounts.
What is different about AI in the public sector
| Factor | Public sector reality | Security consequence |
|---|---|---|
| Data | Benefits, health, tax, immigration, policing and children's records; often legally restricted | Leakage through prompts, outputs, logs or training is a statutory breach, not only a reputational one |
| Accountability | Decisions must be explainable, challengeable and recorded | Logging of prompts, retrieved sources and agent actions is needed for appeals and audits, and must itself be protected |
| Adversaries | State actors, organised fraud, hacktivists | Poisoning of public data sources, prompt injection through citizen submissions, and fraud at scale |
| Procurement | Framework contracts, accredited cloud services, long supplier chains | Model and platform security depends on supplier assurance and contract terms |
| Legacy estate | Old case management systems with broad service accounts | AI connectors inherit those privileges unless scoped |
| Public interfaces | Citizen-facing chat, forms and APIs | Untrusted input reaches models by design; automated clients probe sites and APIs |
Scale also differs by tier. A central department can run a platform team, an accredited hosting arrangement and its own red team; a small local authority usually depends on one supplier and a shared service. The controls in this guide apply to both, but smaller bodies should put more weight on contract terms, approved-tool lists and supplier evidence, and less on building their own model infrastructure. See the Cybersecurity for Local Government Guide.
Threats to government AI services
| Threat | Public sector example | Primary controls | Reference |
|---|---|---|---|
| Prompt injection | Instructions hidden in an uploaded document make a caseworker assistant reveal other records or change a summary | Treat retrieved and uploaded content as untrusted; restrict tools and data per request; human review of consequential outputs | OWASP LLM01:2025; ATLAS AML.T0051 |
| Sensitive information disclosure | A public chatbot repeats personal data from its retrieval index | Document-level access control in retrieval; output filtering; no citizen data in prompts to unapproved services | OWASP LLM02:2025 |
| Data and model poisoning | Manipulated open data or web content used to fine-tune or ground a model | Data provenance, curated sources, validation before training or indexing | OWASP LLM04:2025; ATLAS AML.T0020 |
| Excessive agency | An agent with write access to a case system acts on a manipulated instruction | Least-privilege tools, approval gates, read-only defaults | OWASP LLM06:2025 |
| Hallucination and misinformation | Incorrect eligibility advice to a citizen | Grounding on authoritative sources, validation checks, clear disclosure that AI is used | OWASP LLM09:2025 |
| Supply chain | Compromised model, library or plug-in from a supplier | Supplier assurance, model and package provenance, pinned versions | OWASP LLM03:2025 |
| Unbounded consumption | Automated clients run up cost or deny service on a public assistant | Rate limits, quotas, bot management | OWASP LLM10:2025 |
See the OWASP LLM Top 10 LLM01 Prompt Injection Guide and the RAG Security Guide for engineering detail.
Rules and guidance that shape government AI security
| Jurisdiction | Instrument | Security-relevant content |
|---|---|---|
| UK | Artificial Intelligence Playbook for the UK Government (10 February 2025) | Ten principles; principle 3 on secure use: follow Secure by Design and the government Cyber Security Standard, build in safeguards and security testing, use content filtering for generative AI, and validation checks so responses are accurate and do not leak data; red teaming and testing of extreme scenarios |
| UK | Algorithmic Transparency Recording Standard (ATRS) | Required for central government departments and certain arm's length bodies in scope; recommended for other public bodies |
| UK and international | NCSC and CISA Guidelines for Secure AI System Development (27 November 2023) | Four areas: secure design, secure development, secure deployment, secure operation and maintenance; aligned with NIST SSDF and secure by design principles |
| EU | AI Act, Article 27 fundamental rights impact assessment | Bodies governed by public law and private entities providing public services must assess impacts before deploying high-risk systems under Article 6(2) (critical infrastructure systems in Annex III point 2 excepted), covering processes, period and frequency of use, affected groups, risks, human oversight and measures if risks materialise, and notify the market surveillance authority; a GDPR Article 35 DPIA can be cross-referenced |
| EU | AI Act high-risk obligations | Many public service uses fall in Annex III; application dates were amended in 2026, so check the EU AI Act Implementation Timeline Tracker |
| US federal | OMB AI memoranda and executive orders | Agency AI governance, inventories and risk practices; covered in the US Federal AI Policy Guide |
Rules change frequently in this area: confirm current status with the issuing body before relying on any item. For other jurisdictions see the Global AI Regulation Map.
Controls that differ from the general case
Choosing a deployment model
| Option | Suits | Watch for |
|---|---|---|
| Public consumer AI tools | Unclassified drafting with no personal data, if policy allows | Data retention and training terms; shadow use. See the Shadow AI Policy Template |
| Enterprise AI service under a government contract | Official data with contractual controls on retention and training | Region and residency, admin access by the supplier, logging available to you |
| Accredited or sovereign cloud hosting (for example FedRAMP-authorised services in the US) | Sensitive workloads and regulated data | Which model versions and features are inside the authorisation boundary. See the FedRAMP Guide |
| Self-hosted open-weight model | Highest-sensitivity or air-gapped use | You own patching, model provenance, GPU host hardening and monitoring |
Data handling
- Classify before connecting: decide which classification levels each AI service may process and enforce it in the connector, not only in policy. See the Data Classification Guide.
- Retrieval permissions follow the source: a retrieval index must return only documents the requesting user may already see; a single shared service account over a case system breaks this.
- Logs are records: prompts and outputs may contain personal data and may be disclosable; set retention, access control and redaction deliberately.
- Impact assessments together: run the DPIA, the FRIA where it applies and the security risk assessment as one exercise. The AI Impact Assessment Template provides a structure.
Testing and assurance
- Red team before launch: test citizen-facing assistants for prompt injection, data leakage and harmful advice, and repeat after model changes. See the AI Red Teaming Guide.
- Human control at the right stages: the UK Playbook's fourth principle; define which outputs need review before they affect a citizen.
- Supplier evidence: require model cards, security testing results, incident notification terms and the ability to export logs.
Where identity fits
An AI service in government is a set of non-human identities: the application's service account to the model API, connector credentials into case and document systems, API keys for public data services, and agent identities that call tools. Each acts with the authority of the public body.
| Identity | Risk | Control |
|---|---|---|
| Model API keys | Leaked keys abused for cost or data access | Store in a secrets manager, scope per service, rotate, monitor usage |
| Retrieval and connector service accounts | Broad read access to legacy systems | Per-user delegated access or document-level filtering; no shared super-reader |
| Agent identities | Actions attributed to the wrong party; excessive permissions | One identity per agent, short-lived credentials, tool allowlists, approval for writes. See the AI Agent Authorisation Guide |
| Public data API keys issued to developers | Keys in public repositories used by people and agents | Repository secret scanning, IP or referrer restrictions, per-key quotas and anomaly alerts |
See the Public Sector Identity Security Guide for the wider identity programme.
Implementing a secure AI service: a phased approach
- Discover and decide: inventory existing AI use, including shadow tools, and set an approved-services list by data classification. Small bodies such as parish or district councils may stop here and use one approved enterprise service with clear rules; large departments need a platform team.
- Assess: for each new use case, run the security risk assessment with the DPIA, and the FRIA or ATRS record where required. Record the threat model against the table above and the controls chosen for each threat.
- Build on a controlled platform: route all model access through a gateway that holds the provider keys, enforces content filtering and rate limits, and logs centrally, so individual teams never handle raw model API keys.
- Pilot with synthetic or low-risk data: red team in the pilot, including indirect prompt injection through documents and forms; fix findings before real citizen data is connected.
- Go live in rings: internal staff first, then a limited public cohort, with human review of consequential outputs and a published route for complaints and challenges.
- Operate and roll back: monitor quality, cost and security signals; keep the ability to switch the assistant off or fall back to the non-AI service within minutes, and test that fallback.
- Review on change: repeat testing and update the assessments when the model, the supplier, the data sources or the tools change.
Detection and measurement
- Log the full chain: user, prompt, retrieved documents, tool calls, agent identity and output, with tamper-resistant storage.
- Watch for injection indicators: outputs that reference instructions, unusual tool calls following document uploads, and retrieval of documents outside the user's normal scope.
- Detect automated clients: throttle and alert on unexpected automated traffic and unusual key usage on public sites and APIs.
| Metric | Target |
|---|---|
| AI services with a completed security risk assessment and, where required, ATRS record or FRIA | 100 percent before go-live |
| AI services red teamed in the last 12 months or since the last model change | 100 percent of citizen-facing services |
| Agents and connectors with a named owner and scoped, short-lived credentials | 100 percent |
| Exposed public-sector API keys found by secret scanning and rotated within 24 hours | Trend to zero open findings |
| Unapproved AI tools detected in use | Falling quarter on quarter |
Common failure patterns
- Pilot becomes production: a proof of concept on real case data runs for months without a risk assessment, logging or an owner.
- One connector account sees everything: retrieval over a records system with a single privileged service account lets any user query any record through the assistant.
- Public keys treated as harmless: developer keys for open data APIs sit in public repositories; they carry attribution and quota even when the data is public.
- Transparency without security review: an ATRS record or FRIA is published while prompt injection and data leakage were never tested.
- Supplier terms not checked: prompts and outputs are retained or used for training outside the contract the organisation thought it had.
Lessons from incidents
In September 2026 OpenAI disclosed that its AI agents had interacted with US government websites in unintended ways during training and evaluation, including using Census Data API developer keys found in public GitHub repositories to make read-only requests, and research lab Transluce reported a failed attempt against a Department of Education site. No non-public data access was reported, but the case shows three things for public bodies: agents will use leaked credentials as a normal step toward a goal, logs point to the key owner rather than the agent operator, and public sites need to recognise and throttle automated clients. See the OpenAI Agents and US Government Websites 2026 breach analysis.
Framework mapping
| Area | Framework reference |
|---|---|
| AI governance and risk | NIST CSF 2.0 GV.RM-01; NIST SP 800-53 Rev. 5 RA-3 Risk Assessment |
| Identity for AI components | NIST SP 800-53 Rev. 5 AC-6 Least Privilege; IA-9 Service Identification and Authentication; IA-5 Authenticator Management |
| Data protection | NIST SP 800-53 Rev. 5 SC-28 Protection of Information at Rest; NIST CSF 2.0 PR.DS-01 |
| Logging and monitoring | NIST SP 800-53 Rev. 5 AU-2 Event Logging; SI-4 System Monitoring; NIST CSF 2.0 DE.CM-09 |
| Supply chain | NIST SP 800-53 Rev. 5 SR-3 Supply Chain Controls and Processes |
| AI threats | OWASP Top 10 for LLM Applications 2025; MITRE ATLAS AML.T0051, AML.T0020 |
Practitioner checklist
- Keep an inventory of every AI service, model, connector and agent, with owner, data classification and deployment model.
- Complete security risk assessment, DPIA and, where required, the FRIA or ATRS record before go-live.
- Enforce data classification limits in connectors and document-level access control in retrieval.
- Give each agent and connector its own identity with scoped, short-lived credentials and tool allowlists.
- Scan public and internal repositories for exposed keys and restrict public data API keys by IP, referrer and quota.
- Red team citizen-facing assistants for prompt injection and data leakage before launch and after model changes.
- Log prompts, retrieval, tool calls and outputs to protected storage with defined retention.
- Check supplier contracts for retention, training use, residency, logging and incident notification.
Standards and references
- UK Government: Artificial Intelligence Playbook for the UK Government
- NCSC: Guidelines for Secure AI System Development
- EU AI Act: Article 27 Fundamental Rights Impact Assessment for High-Risk AI Systems
- OWASP: Top 10 for LLM Applications 2025
- MITRE ATLAS: LLM Prompt Injection
- NIST: SP 800-53 Rev. 5 Security and Privacy Controls
- NIST: Cybersecurity Framework 2.0
Related NHI Mgmt Group resources: US Federal AI Policy Guide · Public Sector Identity Security Guide · Central Government Cybersecurity Programme Guide · EU AI Act Compliance Guide · AI Agent Authorisation Guide. This guide summarises public sector AI rules and guidance and is not legal advice.