Generative artificial intelligence learns patterns from training data and then produces new text, images, audio, or code that matches those patterns. Security teams review what data it can access, how prompts are handled, and whether outputs can leak secrets, reproduce protected content, or be manipulated by malicious instructions.
How generative AI learns patterns, not facts
Generative AI is trained on large collections of examples, then learns statistical relationships in that data. It does not store knowledge the way a database does. Instead, it estimates what token, pixel, sound, or code fragment is most likely to come next based on the patterns it has learned. That is why the same model can produce fluent output in many styles, but still make confident mistakes.
The practical implication is that the model’s strength is pattern synthesis, not grounded truth. It can generalise across many examples, but it may also imitate bias, repeat training-data artefacts, or produce plausible-looking output that is not actually correct. That matters when teams use it for drafting, summarising, coding, search augmentation, or decision support.
Because the model is optimising for likely continuation, not verification, the surrounding system has to decide when output is acceptable. That means the quality of prompts, retrieval data, guardrails, and human review can matter as much as the model itself.
What happens between the prompt and the output
At runtime, a prompt is broken into tokens and passed through the model, which transforms the input into a probability distribution over possible next tokens. Each generated token becomes part of the context for the next step, so the output is built iteratively. For images and audio, the mechanism differs in detail, but the same idea holds: the model generates something that matches learned patterns and the current input.
This is why small changes in wording can produce noticeably different answers. Prompt framing, system instructions, conversation history, and any retrieved context all influence the distribution the model uses. If a model is connected to tools or external data, the surrounding orchestration layer also becomes part of the security boundary, because the model may act on information it should not expose or misuse.
For teams building applications, that means the model is only one component in a larger control problem. You need to understand what information enters the prompt, what gets retained in memory, what is returned to the user, and which downstream systems the model can reach.
Why security teams focus on data, prompts, and output handling
Security review usually centres on three questions: what data the model can access, how prompts are handled, and how outputs are validated. Those questions map to the main failure modes: sensitive data leakage, manipulated instructions, unsafe tool calls, and output that looks authoritative but is wrong or harmful. A useful starting point is the LLM Security Guide, which covers the attack surface and the controls that matter most in practice.
Access matters because generative systems are often placed close to internal knowledge, documents, tickets, code, and business workflows. Prompt handling matters because untrusted user input can be crafted to override intended behaviour or exfiltrate hidden context. Output handling matters because the model can reproduce secrets, copyrighted or protected material, or unsafe instructions unless the application filters, constrains, or reviews the result.
That is also why testing should not stop at “does it answer well.” Teams should check whether it reveals hidden instructions, follows malicious prompts, leaks retrieved content, or produces output that should not be executable as-is. Good security practice treats the model as a probabilistic component that needs containment, monitoring, and validation.
Risk and Threat Considerations
Generative AI introduces exposure when a model is given broader access than it needs or when untrusted inputs can influence its behaviour. The main risks are data leakage, prompt injection, output manipulation, and unintended disclosure of training or retrieved content. A model that is useful for broad content generation can become a security problem if it is allowed to see sensitive material without strong boundaries.
Failure mechanism: The system accepts hidden or hostile instructions through prompts, retrieved documents, or tool outputs, then treats them as part of the task and may reveal context, call unintended functions, or generate unsafe content.
Impact: Organisations can expose confidential data, create unreliable business decisions, allow malicious automation, or spread convincing but incorrect output into downstream workflows.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP API Security Top 10 addresses the attack and risk surface, while NIST AI RMF, NIST SP 800-53 Rev 5 and OWASP ASVS set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI Risk Management Framework | Generative AI use requires risk governance across data, prompting, and output handling. |
| Recommendation — Apply AI RMF practices to govern model use, testing, and ongoing risk monitoring. | ||
| NIST SP 800-53 Rev 5 | AC-6 — Least Privilege | GenAI systems should only access the data and tools needed for their task. |
| SI-10 — Information Input Validation | Prompt and retrieved-content handling depend on validating untrusted input before use. | |
| Recommendation — Limit model-connected services to the minimum access required for each workflow. Validate user prompts and retrieved content before they influence model behaviour. | ||
| OWASP ASVS | V16 — Security Logging and Error Handling | GenAI applications need traceability for prompts, tool use, and suspicious outputs. |
| Recommendation — Log model inputs, outputs, and tool actions so abuse and leakage can be investigated. | ||
| OWASP API Security Top 10 | API8 — Security Misconfiguration | Model endpoints and connected tools can leak data or overexpose functions when misconfigured. |
| Recommendation — Harden model-facing APIs and tool integrations against overexposure and unsafe defaults. | ||
Practitioner Guidance
What to prioritise: Start by classifying the model’s inputs and outputs. Identify what data it can read, what it can return, and where human review is mandatory before the output reaches a user, system, or automated workflow.
What to verify: Test the full application path, not just the model prompt. Confirm that retrieved documents, system instructions, tool calls, and final output are each constrained separately, because a control that protects one layer may leave another exposed.
Practitioner takeaway: Generative AI is safest when it is treated as a pattern-producing component inside a controlled workflow, not as a trusted source of truth or authority.
Related resources from NHI Mgmt Group
- What happens when security teams rely on generative AI for external attack surface work without human review?
- How should organisations make artificial intelligence safer without treating it as a simple safe versus dangerous choice?
- How should security teams use cybersecurity podcasts as part of ongoing threat intelligence and awareness work?
- What is the difference between data intelligence and artificial intelligence in analytics programmes?
Deepen Your Knowledge
Free weekly newsletter
Subscribe to the NHI & AI Identity Journal
The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.
Bonus 33% off our NHI Course when you subscribe.
Reviewed and updated by the NHIMG editorial team on October 20, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org