What breaks when organisations rely on traditional application security alone to protect GenAI platforms?

Traditional application security often misses the dynamic layer where prompts, agents, and connected business apps interact. If controls stop at perimeter checks or code review, attackers can exploit promptware, misuse trusted integrations, or trigger unsafe actions through legitimate interfaces. Effective protection needs continuous visibility, behavioural monitoring, and controls designed for AI-driven workflows.

Why This Matters for Security Teams

Traditional application security assumes the risky part of the system is the code path, the perimeter, or the deployed service. GenAI platforms break that assumption because the most dangerous behaviour often emerges after authentication, inside prompts, connectors, agents, and model-driven workflows. A system can pass code review and still be unsafe if it can read sensitive context, call business systems, or execute actions on behalf of users. That is why the control problem shifts from static application hardening to runtime governance, behavioural visibility, and identity-aware enforcement. Current guidance from the NIST Cybersecurity Framework 2.0 and the NIST AI 600-1 GenAI Profile points toward continuous monitoring and AI-specific risk treatment, not one-time application sign-off. NHIMG research on the OWASP Agentic Applications Top 10 shows why this matters: the attack surface expands as soon as an app can reason, retrieve, and act. In practice, many security teams discover the gap only after a trusted integration has already been used to move data or trigger an unsafe downstream action.

How It Works in Practice

Traditional AppSec controls still matter, but they are incomplete for GenAI platforms because they rarely inspect what happens at runtime when a prompt changes, a tool is invoked, or an agent chains actions across services. The practical failure mode is not just injection. It is an authenticated system making an untrusted decision. That means security teams need controls that understand intent, context, and downstream effect, not just request validation and code quality. NHIMG cases such as the McKinsey AI platform breach and the CrewAI GitHub Token Leak illustrate how trusted AI workflows can expose sensitive data or abused credentials once the platform starts interacting with real systems.

• Use runtime policy checks for every model call, tool invocation, and connector action.
• Treat prompts, retrieved context, and outputs as governed data flows, not just text.
• Scope secrets and API tokens to the minimum task, with short TTLs and automatic revocation.
• Separate user intent from model execution so the model cannot silently expand privilege.
• Log agent decisions, tool calls, and data access paths for investigation and containment.

This is where workload identity and policy-as-code become more useful than traditional role-only controls. If an agent can access CRM data, file stores, and ticketing systems, the decision should be made at request time based on the task, context, and trust level, not on a static role name alone. These controls tend to break down when legacy applications expose broad API permissions because the GenAI layer inherits privileges faster than teams can redesign them.

Common Variations and Edge Cases

Tighter runtime control often increases latency, operational overhead, and integration complexity, so organisations must balance safety against developer velocity and user experience. That tradeoff is real, especially in environments that rely on many plugins, external APIs, or human-in-the-loop approvals. Best practice is evolving, but there is no universal standard for how much autonomy is acceptable in every workflow. For some systems, prompt filtering and output moderation may be enough; for others, especially where agents can execute transactions, stronger controls are required.

One common edge case is shadow AI, where employees connect sanctioned data to unsanctioned tools. Another is agent sprawl, where multiple autonomous components share the same credentials and make attribution difficult. A third is retrieval-augmented generation that looks safe at the application layer but leaks sensitive content through the retrieval layer. The State of Secrets in AppSec is relevant here: when secrets management is fragmented, organisations lose the ability to answer which credential was used, by whom, and for what purpose. The right response is not only stronger AppSec, but tighter governance for identities, secrets, and AI-specific runtime decisions. Where the platform cannot enforce per-action policy or isolate credentials per agent, traditional application security stops being the control boundary.

Related resources from NHI Mgmt Group

• What breaks when organisations rely on EDR alone for browser security?
• What breaks when organisations rely on severity scores alone for application triage?
• What breaks when organisations rely on thirty-day remediation targets for application security?
• What breaks when organisations rely only on post-ingest application security scanning to stop malicious packages and supply-chain compromise?