Generative artificial intelligence matters because it can create text, code, images, and other outputs that may be useful for automation or abuse. Security teams need controls for data exposure, prompt injection, output validation, access to tools, and model governance. Without oversight, it can generate misleading content, leak sensitive data, or carry out unsafe actions.
Why generative AI changes the security workload
Generative AI expands the security surface because it can produce convincing content at scale, not just consume it. That matters when the same system can draft phishing text, generate code, summarise sensitive data, or recommend actions through connected tools. Security teams have to treat it as both a productivity layer and a potential abuse amplifier, with controls that match its speed, reach, and ambiguity.
The practical shift is that security review can no longer stop at whether the model “works.” Teams also need to know what data it can see, what it is allowed to say, and what it can trigger. A model that is useful for analysts may still be unsafe if it can be steered into revealing secrets, inventing falsehoods, or taking actions outside the intended workflow.
Where the main exposure comes from
Most of the risk sits in the interfaces around the model, not only inside the model itself. Prompt injection, unsafe tool access, and weak output handling can turn ordinary user inputs into a path for data exposure or unauthorised actions. Security teams should also expect the model to be wrong in plausible ways, which makes output validation and human review important for any decision that affects access, content integrity, or response actions.
This is why generative AI often creates compound risk. A single prompt can influence what data is retrieved, what action is proposed, and what text is shown to an operator. If the model has access to internal systems, the issue is no longer just content quality, it becomes a control and trust problem.
What security teams should govern before rollout
Security teams need explicit rules for data exposure, tool permissions, logging, and model change control. That includes deciding which data classes may be sent to a model, which prompts or outputs must be retained, and which actions require approval before execution. A good deployment assumes that the model will sometimes produce inaccurate, incomplete, or manipulative output, so governance must define where the human checkpoint sits.
When the model is connected to internal systems, access should be narrowly scoped and reviewed like any other privileged integration. Teams should separate experimentation from production use, and they should know who owns model updates, system prompts, retrieval sources, and red-team findings. Without that ownership, the risk is not just misuse, but slow drift into unsafe usage patterns.
Risk and Threat Considerations
Generative AI introduces both misuse risk and trust-boundary risk. Attackers can use it to scale social engineering, write malware-like content, or search for weak points in workflows, while defenders can accidentally expose sensitive data or let untrusted instructions steer connected tools.
Failure mechanism: Prompt injection, data leakage, hallucinated output, and overbroad tool permissions can combine so that a model follows attacker-controlled instructions or reveals information it should not expose.
Impact: The result can be compromised confidentiality, unsafe automated actions, fraud support, incident-response confusion, or a broader loss of trust in security decision-making.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP API Security Top 10, OWASP Agentic AI Top 10, MITRE ATT&CK and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST AI RMF sets the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI Risk Management Framework | Generative AI governance, testing, and risk controls directly shape this subject. |
| Recommendation — Apply the AI RMF to manage GenAI risks across governance, mapping, measurement, and controls. | ||
| OWASP API Security Top 10 | API5 — Broken Function Level Authorization | Model-connected APIs can expose actions the user should not be able to invoke. |
| Recommendation — Enforce function-level authorization on every model-triggered API action. | ||
| OWASP Agentic AI Top 10 | ASI02 — Tool Misuse | GenAI becomes risky when it can misuse connected tools or services. |
| ASI03 — Identity & Privilege Abuse | Connected agents or assistants can overstep intended authority if not governed. | |
| Recommendation — Constrain tool access and require approval for high-impact tool use. Bind agent actions to explicit privilege boundaries and review elevated access paths. | ||
| MITRE ATT&CK | T1204 — User Execution | GenAI output can be used to induce users to execute malicious or unsafe actions. |
| Recommendation — Map GenAI-enabled social engineering to user-execution detections and response playbooks. | ||
| OWASP Non-Human Identity Top 10 | NHI-02 — Secret Leakage | GenAI systems can expose secrets through prompts, logs, or generated output. |
| Recommendation — Protect model prompts, logs, and connectors from leaking secrets or tokens. | ||
Practitioner Guidance
What to prioritise: Start with the use cases that can touch sensitive data or invoke tools, not the low-risk drafting tasks. The highest-value control work is usually in retrieval sources, prompt handling, tool permissions, and output review, because those are the places where a useful assistant becomes an execution path.
What to verify: Confirm that the model cannot reach data or systems it does not need, that high-impact outputs are checked before use, and that prompt, context, and tool events are observable enough for investigation. If you cannot explain why a model had access to a resource, the control design is incomplete.
What good looks like: The model can assist analysis or drafting, but it cannot silently cross from suggestion to action. Sensitive tasks remain attributable, access remains bounded, and the team can show which instructions, sources, and permissions were in force at the time of use.
Practitioner takeaway: Treat generative AI as a security-adjacent decision engine, not a content tool with extra features. The core question is whether its outputs and side effects are constrained enough that speed does not outrun control.