What breaks when organisations rely on static scans for AI security?

Why This Matters for Security Teams

Static scanning still has value for code hygiene, dependency review, and catching obvious insecure patterns, but it is not enough for AI systems that change behaviour at runtime. The core issue is that model risk is often expressed through interaction, not source code. Prompt injection, tool abuse, retrieval poisoning, and unsafe output chaining only become visible when the system is exercised in a live context. That makes AI security closer to an operational assurance problem than a one-time code review problem.

This is where teams frequently overstate confidence. A clean scan report can create the impression that the system is safe, even when the model can be manipulated by user input, malicious context, or fragile orchestration logic. Guidance from NIST AI Risk Management Framework is useful here because it treats AI risk as a lifecycle issue, not just a build-time issue. For agentic systems, the question is not only whether the code is clean, but whether the agent can be induced to take harmful actions with legitimate tools and permissions. In practice, many security teams discover that their static checks were never testing the real failure path until a model has already exposed data, executed an unsafe action, or amplified a bad instruction.

How It Works in Practice

Effective AI security usually combines static analysis with runtime testing, policy enforcement, and adversarial evaluation. Static scans can still flag hard-coded secrets, unsafe libraries, weak input validation, and exposed endpoints. But they do not simulate the full behaviour of an LLM or agent across prompts, retrieval sources, tool calls, and post-processing steps. That is why testing must include prompt injection attempts, malformed context, jailbreak-style inputs, and malicious retrieval content.

For agentic workflows, the security question expands from “does the code look safe?” to “what happens when the model decides to act?” That includes whether the model can call tools it should not access, whether output is validated before execution, and whether there are guardrails around data egress. The CSA MAESTRO agentic AI threat modeling framework is useful because it pushes teams to map threats across orchestration, memory, tools, and trust boundaries.

• Use static scans to catch insecure implementation details, not to certify model safety.
• Run red-team tests against prompts, retrieval pipelines, and tool permissions.
• Validate outputs before they trigger downstream actions, especially in automation.
• Monitor runtime logs for abnormal tool use, repeated refusal patterns, and data leakage.
• Apply least privilege to model-connected services, APIs, and credentials.

Where relevant, this should be paired with control testing from OWASP Top 10 for LLM Applications and adversarial scenario mapping from MITRE ATLAS, because attack paths in AI systems often emerge from interaction effects that static analysis cannot model. These controls tend to break down when the model has broad tool access and no human approval step because the dangerous action happens after the scan has already concluded.

Common Variations and Edge Cases

Tighter pre-deployment review often increases engineering overhead, requiring organisations to balance build speed against confidence in runtime behaviour. Best practice is evolving, and there is no universal standard for how much static testing is “enough” for an AI application. For narrow, read-only systems, static analysis may catch a meaningful share of implementation risk. For agentic systems, however, the highest-impact failures usually come from context, permissions, and orchestration rather than source code alone.

Edge cases matter. A retrieval-augmented system may look well secured until an attacker poisons the knowledge base. A chatbot may pass scan-based review yet still leak sensitive data when prompted with carefully shaped context. A workflow agent may inherit safe code but unsafe access to email, ticketing, or cloud APIs. In those cases, static scans can confirm that the application was built correctly, but they cannot prove the system will behave correctly under pressure. Current guidance suggests treating static scanning as one input to a broader assurance program that includes runtime policy checks, red teaming, and monitoring for abnormal model actions.

When the system is governed as part of a larger AI estate, Anthropic Project Glasswing is a useful reference point for understanding how dangerous behaviour can emerge from deployed model interactions rather than from code defects alone. The practical takeaway is simple: static scans reduce known implementation flaws, but they do not replace attack-surface testing for live prompts, tool use, or data flow.

Related resources from NHI Mgmt Group

• What breaks when organisations rely on manual data classification for AI security?
• What breaks when organisations rely on static permissions for enterprise AI search?
• What breaks when organisations rely on legacy DLP for AI workflows?
• What breaks when organisations rely only on observability for AI governance?