Generative AI

Expanded Definition

Generative AI refers to systems that produce new content from prompts, such as text, code, images, or audio. In NHI security, the term matters because the model itself usually does not hold durable privilege, but the surrounding application, connectors, and prompt handling can expose secrets, data, and policy violations. That makes the security boundary less about the model alone and more about the full workflow that injects context, routes outputs, and executes follow-on actions. The NIST AI 600-1 Generative AI Profile treats these systems as a distinct risk class because output quality, data provenance, and misuse controls need explicit governance rather than generic application security alone. Guidance varies across vendors on how much autonomy counts as “generative AI” versus agentic ai, so the distinction should be made carefully when a model can also call tools or trigger workflows. The most common misapplication is treating a content model as harmless because it is “only generating text,” which occurs when prompt inputs, retrieved context, or copied secrets are not governed as sensitive attack surface.

For related breach context, see the DeepSeek breach and the Microsoft Azure OpenAI service breach.

Examples and Use Cases

Implementing generative AI rigorously often introduces review overhead and access constraints, requiring organisations to weigh faster content production against stronger controls on prompts, outputs, and embedded data.

• A developer uses a code assistant to draft functions, but the organisation blocks source uploads that include secrets, aligning with the secrets risk patterns discussed in The State of Secrets in AppSec.
• A support team deploys a chat assistant for customer replies, while human review is required for regulated disclosures, legal claims, and account changes.
• An internal knowledge bot retrieves policies and summaries, but retrieval scopes are limited so the model cannot expose confidential records from unrelated repositories.
• A marketing team generates campaign copy, yet the workflow scans outputs for brand, compliance, and copyright issues before publication.
• A security team uses a GenAI summariser to triage alerts, following the risk framing in the NIST AI 600-1 Generative AI Profile.

In practice, these use cases work best when prompts, retrieved context, and output destinations are all treated as governed data paths rather than informal productivity shortcuts.

Why It Matters in NHI Security

Generative AI becomes an NHI issue whenever it is connected to systems that contain credentials, tokens, or sensitive business data. The model may not be autonomous, but it can still reproduce secrets, reveal restricted context, or amplify bad instructions into unsafe outputs. NHI teams care because these tools often sit beside service accounts, API keys, and identity-aware workflows that can be abused through prompt injection, overbroad connectors, or poor output filtering. NHIMG research shows that 43% of security professionals are concerned about AI systems learning and reproducing sensitive information patterns from codebases, which is especially relevant when code assistants are trained or prompted on internal repositories. That concern is not abstract when one incident can expose many assets at once, as seen in the DeepSeek breach and the Microsoft Azure OpenAI service breach. Organisations typically encounter this term’s operational importance only after a model has already leaked data, copied a secret into an output, or sent a sensitive response to the wrong user, at which point generative AI governance becomes unavoidable to address.

Related resources from NHI Mgmt Group

• AI Agent Authentication
• Shadow AI
• What is Agentic AI and how does it differ from traditional generative AI?
• How should security teams govern API keys used for generative AI access?