AI security is repeating the old rule-based failure pattern

At a glance

What this is: Lakera argues that AI security is replaying the old rule-based failure pattern, where static guardrails cannot keep pace with dynamic model behaviour.

Why it matters: This matters because IAM and security teams need controls that adapt to runtime behaviour, not just policies that look comprehensive on paper.

Context

AI security is a governance problem when the system being defended changes its behaviour at runtime. Static filters, prompt rules, and manual test cases can cover obvious cases, but they do not scale to open-ended model interactions or novel attack phrasing.

Lakera frames the issue as a repeat of an older technology shift: rule-based systems worked until the problem became too variable for handcrafted logic. For AI security programmes, the lesson is that assurance must move closer to detection, adaptation, and behaviour-based control rather than exception handling.

Key questions

Q: What breaks when AI security relies on static prompt rules?

A: Static prompt rules break when the same unsafe intent can be expressed through new wording, context, or sequencing. They catch known patterns, but they do not generalise to behaviour that changes at runtime, so teams end up chasing exceptions instead of governing the real attack surface.

Q: Why do static guardrails create false confidence in AI security?

A: They can block obvious prompts while leaving rephrased, indirect, or chained attacks untouched. That creates the illusion of coverage because the control appears active, but the security model has only proven it can handle the examples already anticipated by the team.

Q: What are the signs that AI assurance is too narrow?

A: The clearest sign is when teams only test model accuracy or prompt behaviour while ignoring data integrity, provenance, access control and monitoring. That usually means the programme is validating outputs but not the operational conditions that make those outputs trustworthy. Assurance needs to cover the full path from data input to deployment.

Q: How can security teams tell whether AI lifecycle controls are working?

A: They should look for evidence that access requests, policy enforcement, and usage visibility are centrally recorded and current. If those signals are fragmented across platforms, the programme may be documenting governance rather than enforcing it. Continuous traceability is the practical test.

Technical breakdown

Why static prompt filters fail against dynamic model behaviour

Static prompt filters and handcrafted guardrails depend on known patterns. That works only when the threat surface is stable enough to enumerate, but AI interactions are open-ended, adversarial, and easy to rephrase. A blocked prompt injection does not end the problem, because the same intent can be expressed through different wording, context, or sequence. This is the same structural weakness that broke rule-based vision and language systems: the control logic tries to anticipate every variant instead of learning what unsafe behaviour looks like. In practice, this turns defence into a maintenance treadmill.

Practical implication: treat static filters as a narrow control layer, not the security model.

Why manual test cases do not generalise to AI attacks

Manual red-team prompts and fixed test suites capture only the scenarios the team already imagined. AI systems, however, can be attacked through prompt injection, jailbreak variation, context manipulation, and indirect instruction paths that emerge after deployment. The mechanism problem is generalisation: a test case proves one path is blocked, not that the system understands the boundary between safe and unsafe behaviour. In the article’s analogy, the security function is still writing rules for a problem that behaves more like perception and language than like conventional application logic. That makes brittle coverage the default outcome.

Practical implication: use test cases to seed detection logic, not as proof of durable protection.

Why AI-native defence has to learn attack patterns

Lakera’s core point is that security for AI needs the same kind of pattern learning that made modern AI viable in the first place. The point is not to replace all rules, but to supplement them with systems that can recognise suspicious behaviour even when the exact phrasing is new. That matters because the attack surface changes with the model, the prompt, the tool chain, and the user context. AI-native defence shifts the emphasis from enumerating prohibited strings to identifying attack intent, boundary crossing, and abnormal action sequences.

Practical implication: build detection that scores behaviour and context, not just matching text.

Breaches seen in the wild

• 12,000 secrets in LLM training data: Truffle Security found 11,908 live API keys and passwords hard-coded in web pages captured by Common Crawl, a dataset used to train LLMs.
• Microsoft SAS token exposure 2023: An over-permissive Azure SAS token in a Microsoft AI GitHub repo exposed 38TB, including workstation backups and Teams messages, for 3 years.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Rule-based AI security is a control model mismatch, not a tuning problem. The article’s central point is that dynamic model behaviour outpaces handcrafted policy logic by design. That is not a matter of improving the rule set, but of recognising that the control philosophy no longer matches the attack surface. Practitioners should read this as a signal to stop treating prompt filtering as the primary security boundary.

AI security needs runtime assurance because attack variation is part of the threat model. Prompt injection, jailbreaks, and indirect instruction attacks mutate faster than deterministic allowlists can absorb. The useful distinction is between controls that block known strings and controls that infer unsafe intent from behaviour. That shift matters because the latter is the only model that can stay relevant as usage patterns evolve.

Static guardrails create false confidence when they are mistaken for coverage. A system can appear protected because basic prompts are blocked, while the real risk sits in reworded, contextual, or chained interactions. This is the same governance failure pattern seen in earlier rule-based systems, where coverage looked broad until adversaries found what the rules did not model. Security teams should treat apparent control completeness as a validation problem, not an assurance outcome.

AI security programmes will increasingly be judged by adaptation speed, not policy thickness. The article points to a field where the pace of model change punishes controls that need frequent manual rewriting. That does not remove the need for policy, but it does change the yardstick: the real question is whether the security model can absorb new attack shapes without starting from scratch. Practitioners should measure resilience in response time and generalisation, not in the number of written exceptions.

Behavioral detection is the named concept that will separate durable AI security from brittle rule sets. In this context, behavioural detection means identifying unsafe model use by patterns of interaction, not by exact prompt signatures. That concept matters because it aligns the defence with the way attacks actually mutate across sessions, users, and tools. Practitioners should anchor AI security design around behaviour, context, and adaptation rather than static blocklists.

From our research library:

• Only 23% of IT leaders were very confident in their organisation's ability to manage security and governance for GenAI deployments, according to a 2025 Gartner survey of 360 IT leaders.

What this signals

Behavioural detection is the control shift that matters most here: AI security cannot depend on enumerating every forbidden prompt, because the attack surface is shaped by variation, context, and model response. Programme owners should focus on controls that can infer unsafe intent from runtime behaviour rather than from exact text matches.

The governance lesson extends beyond model safety into broader identity security: whenever the threat can mutate faster than the rule set, static policy becomes a lagging indicator. Teams should treat coverage claims as provisional until the control can demonstrate generalisation across unseen variants.

For practitioners

• Replace prompt-only guardrails with behaviour-based detection Use signals such as instruction escalation, repeated boundary probing, tool abuse, and abnormal context shifts to identify unsafe interactions. Keep text rules for obvious blocks, but do not rely on them as the main control plane.
• Build adversarial test coverage for rephrasing and context shifts Expand evaluation beyond fixed jailbreak examples to include paraphrase variation, indirect instruction paths, and chained prompt manipulation. Measure whether controls still hold when the attack intent stays the same but the language changes.
• Separate policy enforcement from safety assurance Treat written policy as governance intent and model-driven monitoring as operational assurance. The former defines acceptable use, while the latter checks whether the system is drifting into unsafe behaviour at runtime.
• Track false confidence from static coverage Review whether your current controls are blocking only the examples you already know. If teams cannot explain how a new prompt pattern would be judged safe or unsafe, the programme is depending on brittle exceptions.

Key takeaways

• AI security breaks down when teams mistake static prompt filters for a durable defence model.
• The article’s core warning is that rule-based control cannot keep pace with changing attack phrasing and context.
• Practitioners should shift toward behavioural detection and runtime assurance rather than counting exceptions caught by handcrafted rules.

Key terms

• Behavioral Detection: A monitoring approach that looks for unusual activity rather than relying only on static inventories. For SaaS integrations, it detects drift in token use, data movement, timing, and endpoint behavior so teams can spot compromise, misuse, or automation that no longer matches its expected pattern.
• Runtime Guardrail: A control applied while an AI agent is operating, not just during configuration or review. Guardrails can block dangerous tool calls, require approval for sensitive actions, or stop data leakage before it reaches systems or users.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Runtime assurance: Runtime assurance is the practice of validating how an application actually behaves after deployment. It matters because configuration, identity flow, and integration state can change security outcomes in ways that source code analysis alone cannot prove.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an identity security programme, it is worth exploring.