GenAI Security works by reducing the risk that generative AI systems leak data, follow malicious instructions or produce harmful outputs. Defenders secure prompts, training data, connectors, access controls and logging, then test for prompt injection, data exfiltration, model misuse and unsafe tool execution across the full application path.
What GenAI Security is actually protecting
GenAI Security is about controlling how a generative AI system sees data, interprets instructions, uses tools and returns output. The practical target is not just the model itself, but the full application path around it, including prompts, retrieved context, connectors, identity, logging and output handling. That is why the controls focus on both misuse resistance and data protection.
Because GenAI systems can act on untrusted text and on connected enterprise data, security has to treat prompt content, retrieved content and tool calls as potential attack surfaces. A secure design assumes that malicious instructions can arrive through users, documents, web content or connected systems, and it verifies each step before the system is allowed to disclose, transform or execute anything sensitive.
GenAI Security also has a strong operational dimension: teams need to know which data the system can reach, which tools it can invoke, and what gets recorded for review. LLM Security Guide is useful here because it frames prompt injection, data leakage and unsafe output handling as part of one defensive surface rather than separate problems.
How the defense layers fit together
The first layer is input control. Prompts, uploaded files, retrieved snippets and external content should be treated as untrusted until they are validated or constrained. This matters because the model may follow the latest or most forceful instruction it sees, even when that instruction is embedded in content that was supposed to be informational only.
The second layer is access control around data and tools. A GenAI application should only be able to retrieve the minimum data needed for the task, and only invoke the tools that task requires. If the system can reach broad data stores or high-impact actions, then a prompt injection event can become a data exposure or a real-world action issue, not just a bad answer.
The third layer is monitoring and traceability. Security teams need logs that show what was asked, what context was retrieved, what tools were called and what output was produced. That evidence is what makes investigation possible when the model behaves oddly, gives an unsafe answer or appears to have been steered by hidden instructions.
These same concerns are reflected in NIST AI 600-1 GenAI Profile, which ties generative AI risk management to governance, provenance, testing and incident handling rather than treating model safety as a purely technical tuning problem.
Where GenAI failures usually show up
Most GenAI security failures are interaction failures. Prompt injection can redirect the system, data exfiltration can occur through responses or tool output, and unsafe tool execution can turn a normal request into an unauthorized action. These issues are especially serious when the model is connected to search, ticketing, messaging, code execution or business systems.
Another common failure mode is over-trust. People may assume the system is merely summarising content when it is actually assembling instructions from several sources and then acting on them. That creates a gap between how the tool is perceived and how it behaves, which is why testing must include adversarial prompts, malicious documents and hostile connector content.
Security also fails when deployment teams focus on the model and ignore the integration layer. A safe base model can still be exposed through weak retrieval rules, overbroad connector permissions, poor session handling or missing output filtering. OWASP API Security Top 10 is relevant because many GenAI systems expose their highest-risk behaviour through APIs and backend services.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP API Security Top 10 and MITRE ATT&CK address the attack and risk surface, while NIST AI 600-1 sets the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI 600-1 | Governance, testing, provenance, incident handling — Generative AI Profile | Directly addresses GenAI governance, testing and incident handling. |
| Recommendation — Apply the GenAI profile to govern retrieval, testing and incident response. | ||
| OWASP API Security Top 10 | API8 — Security Misconfiguration | GenAI apps often expose risk through misconfigured APIs and connectors. |
| Recommendation — Harden API configuration and access paths that mediate GenAI actions. | ||
| MITRE ATT&CK | T1027 — Obfuscated Files or Information | GenAI prompt injection and payload hiding often rely on concealed instructions. |
| Recommendation — Hunt for hidden-instruction delivery patterns in inputs and retrieved content. | ||
Practitioner Guidance
What to prioritise: Start with the parts of the GenAI stack that can cause real loss if they are manipulated, which usually means connector scope, tool permissions, retrieval boundaries and logging before model tuning. If the system can read sensitive records or trigger actions, treat those pathways as the primary control surface.
What to verify: Test the full request path with hostile prompts, poisoned documents and indirect instruction payloads, then confirm whether the system resists disclosure, refuses unsupported actions and records the event in a way analysts can investigate. If you cannot reconstruct what the model saw and did, the control design is incomplete.
Decision rule: If a GenAI feature can access production data or invoke business tools, require explicit approval boundaries and least-privilege access for each integration. If it cannot be bounded that way, keep it in a constrained or read-only mode until the exposure is reduced.
Practitioner takeaway: GenAI Security is effective when the system is treated as an exposed decision path, not a black box, so the main job is to bound what it can see, what it can do and what you can prove after the fact.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Free weekly newsletter
Subscribe to the NHI & AI Identity Journal
The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.
Bonus 33% off our NHI Course when you subscribe.
Reviewed and updated by the NHIMG editorial team on October 18, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org