How do security teams decide whether LLM security testing belongs in AppSec or a separate AI security program?

Why This Matters for Security Teams

The decision is not just an org chart question. When an LLM is embedded in a product, the security failure often shows up as unsafe application behaviour, not as a model defect in isolation. That means prompt injection, data leakage through context, broken output handling, and over-permissive tool access belong in the same risk conversation as API security and application logic. NIST’s NIST AI Risk Management Framework is useful here because it treats AI risk as lifecycle risk, not a one-time model review.

Teams often get this wrong by creating a separate AI review gate that looks at prompts and model choice, while AppSec looks at the surrounding code and neither team owns the combined attack path. That split creates blind spots around context-window abuse, unsafe retrieval, and tool invocation. The more the LLM can read, write, or act on behalf of a user, the more the test plan needs to follow the application path rather than the model boundary. In practice, many security teams encounter LLM risk only after prompt injection has already reached a live workflow, rather than through intentional pre-release testing.

How It Works in Practice

A practical operating model is to keep LLM security testing inside AppSec for product-facing systems, while adding AI specialists for model-specific concerns such as provenance, alignment, and training data integrity. AppSec should own the control surface where the model touches users, data, and downstream tools. AI security can define threat scenarios and test methods, especially for agentic workflows and retrieval-augmented generation. The best practice is evolving, but current guidance suggests using a shared intake and triage model instead of forcing one team to own everything.

For embedded LLM features, testing usually covers:

• Prompt injection resistance in user input, retrieved content, and tool outputs.
• Authorization checks on every tool call and data fetch.
• Output validation for unsafe instructions, secrets exposure, and policy violations.
• Logging and monitoring that preserve evidence without leaking sensitive prompts or context.
• Red-team scenarios that simulate abuse of memory, retrieval, and action-taking capabilities.

Security teams increasingly map these tests to AI-specific guidance such as the OWASP Agentic AI Top 10 and NIST AI 600-1 Generative AI Profile, while using AppSec methods to verify input handling, access control, and secure deployment patterns. Where systems use autonomous actions or tools, the CSA MAESTRO agentic AI threat modeling framework helps teams structure threat paths across orchestration, memory, and external dependencies. These controls tend to break down when the LLM is deployed across multiple microservices and each service team assumes another group owns the prompt, context, or tool boundary.

Common Variations and Edge Cases

Tighter separation between AppSec and AI security can improve specialist depth, but it also increases coordination overhead, so organisations have to balance technical expertise against operational speed. In low-risk use cases, a lightweight AI review may be enough, especially if the model only drafts text and cannot access tools or sensitive data. For higher-risk systems, best practice is evolving toward shared ownership with clear RACI boundaries, because no universal standard says every LLM needs a separate security program.

The edge cases are usually the ones with the highest blast radius: customer-facing agents that can trigger actions, internal copilots that can access proprietary documents, and RAG systems that ingest untrusted sources. In those environments, the relevant question is not “AI program or AppSec?” but “who tests the full chain from user prompt to model output to side effect?” That is also where agentic AI governance starts to intersect with NHI controls, because tool credentials, service identities, and secret handling become part of the LLM attack surface. The current guidance suggests a split only when the AI system is truly standalone and does not influence production application behaviour. For more autonomous or integrated systems, separate review notes are not enough, and unified testing is the safer default. If teams ignore that distinction, the model may pass review while the application still fails in production.

Related resources from NHI Mgmt Group

• How should teams decide whether AI procurement belongs in security governance review?
• How do teams decide whether AI governance belongs in security, privacy, or platform engineering?
• How should security teams decide whether an AI agent gets human or non-human identity?
• How do security teams decide whether to let AI agents automate investigations?