GenAI security matters because generative systems can leak data, produce harmful output, execute unwanted actions, and be manipulated through prompt injection or unsafe integrations. Security teams need controls for identity, data exposure, tool permissions, logging, and human review. Without those controls, a model can become a path for fraud, malware support, or unauthorised disclosure.
How GenAI changes the security team’s job
GenAI turns a model from a passive tool into an active system that can expose data, shape decisions, and trigger downstream actions. That shifts the security team’s focus from only protecting content to governing prompts, outputs, integrations, identity, and approval paths. The control problem is not just what the model knows, but what it can reveal or do.
A useful way to think about the risk is that GenAI collapses boundaries between data handling, application logic, and user interaction. A prompt can carry sensitive context, an output can become an instruction, and a connected tool can turn a hallucination into a real-world action. That is why security teams must treat GenAI as an attack surface, not only as a productivity feature.
For teams building or reviewing controls, the first question is whether the model is allowed to influence anything with business impact. If the answer is yes, then the model’s data inputs, tool permissions, and logging become security-relevant design choices, not optional enhancements. The LLM Security Guide is the most direct internal starting point for the risks, attack paths, and hardening measures that sit around GenAI systems.
Why prompt injection, leakage, and unsafe actions matter
GenAI security matters because the main failure modes are not theoretical. Prompt injection can steer the model away from policy, data leakage can expose confidential material in prompts or outputs, and unsafe integrations can let an apparently harmless response trigger an action outside the user’s intent. The same control gap can therefore create confidentiality, integrity, and fraud exposure at once.
One reason these failures are serious is that the model often sits between people and tools. If the model can search, send, summarise, approve, or execute, then a manipulated instruction may reach systems that were never meant to trust the original prompt. The OWASP Top 10 for LLM Applications Guide is useful here because it maps the common failure classes to practical prevention and testing concerns.
Security teams also need to watch for overreliance. When users treat model output as authoritative, harmful advice can be acted on too quickly, especially in investigations, code generation, customer communications, or financial workflows. The risk is not only that the model is wrong, but that the organisation has allowed model output to behave like a trusted intermediate result.
What security teams should control first
Security value comes from constraining the model’s authority before trying to perfect the model itself. That means tightly defining what data it can see, which tools it can call, what actions require human approval, and how every important interaction is logged. Without those boundaries, GenAI becomes a broad amplifier for existing mistakes in access control and workflow design.
For programme design, maturity matters more than isolated fixes. The AI Security Maturity Model helps teams think in stages, from inventory and visibility through governance and runtime controls, rather than jumping straight to tool-specific hardening. That sequencing is important because teams cannot protect what they have not inventoried, classified, or assigned ownership for.
The other control priority is identity. If a model, agent, or integration can act on behalf of a user or service, then permissions need to be explicit, bounded, and revocable. Security teams should not accept broad standing access just because a workflow is automated; they need to know which identity is acting, what it can reach, and what evidence remains after it acts.
Risk and Threat Considerations
GenAI introduces a combined risk of disclosure, manipulation, and abuse of trust. A prompt can leak sensitive text, a model can produce convincingly wrong guidance, and an integration can convert adversarial input into unauthorized action. That makes GenAI attractive to attackers because it can be used to extract information, support fraud, or extend access through ordinary business workflows.
Failure mechanism: The control boundary fails when prompts, retrieval sources, tool permissions, or output handling are not separated tightly enough, allowing injected instructions or exposed context to influence business systems.
Impact: The result can be data loss, unsafe automation, malware assistance, false approvals, or a broader compromise path that is harder to spot than a direct application exploit.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP API Security Top 10 and OWASP Agentic AI Top 10 address the attack and risk surface, while NIST AI 600-1, NIST SP 800-53 Rev 5 and NIST Zero Trust (SP 800-207) set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP API Security Top 10 | API8 — Security Misconfiguration | GenAI integrations often fail through weak API and tool exposure controls. |
| Recommendation — Harden model-connected APIs and restrict exposed functions to the minimum required. | ||
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | GenAI systems become risky when agents can act with excessive authority. |
| ASI02 — Tool Misuse | The question centers on unsafe tool calls and unintended actions from model output. | |
| Recommendation — Bound agent authority and require explicit approval for high-impact actions. Constrain tool access and validate every action path before execution. | ||
| NIST AI 600-1 | Generative AI Profile | This profile directly addresses governance, testing, provenance, and incident handling for GenAI. |
| Recommendation — Apply GenAI profile guidance to govern content provenance, testing, and incident response. | ||
| NIST SP 800-53 Rev 5 | AU-2 — Event Logging | GenAI security depends on retaining evidence of prompts, tool calls, and outputs. |
| IA-5 — Authenticator Management | GenAI controls depend on managing secrets and credentials used by connected tools and services. | |
| AC-6 — Least Privilege | Model and tool permissions must be tightly limited to reduce blast radius. | |
| Recommendation — Log model interactions and administrative actions with enough detail for reconstruction. Rotate and protect credentials used by model integrations and tool backends. Grant each GenAI component only the access needed for its approved function. | ||
| NIST Zero Trust (SP 800-207) | Zero Trust Architecture | GenAI systems need explicit verification of identity, context, and access before action. |
| Recommendation — Verify each request and segment model access from sensitive systems. | ||
Practitioner Guidance
What to prioritise: Start with the workflows where a model can see sensitive data or trigger an action. Those are the places where a prompt mistake becomes a real security event, so they deserve the strictest review and the strongest approval controls.
What to verify: Confirm that logging captures the prompt, relevant retrieved context, tool calls, and the final output, and that someone can reconstruct what the model was allowed to do. If you cannot explain the action path after the fact, the control surface is too loose.
Decision rule: If a GenAI feature can disclose confidential information or initiate an external action, require explicit authorization boundaries and human review for the highest-risk steps. If it cannot, keep it read-only and reduce its trust level accordingly.
Common mistake: Treating the model as the primary control object. The stronger practice is to govern the surrounding system, especially data scope, integration scope, and approval scope, because that is where most material failures emerge.
Practitioner takeaway: GenAI security is really about limiting what the model can see, influence, and execute, then proving those limits are observable and enforceable in practice.