What are the signs that AI security controls are too dependent on frontier model defaults?

A key warning sign is when an organisation has deployed customer-facing or internal AI applications but has not added AI-specific security protocols beyond default model settings. Another indicator is overreliance on basic system prompt guardrails while ignoring attack testing, confidential data exposure, and tool access restrictions. In that posture, a single prompt path can become a repeatable control gap.

Why Default Model Settings Are Not a Security Strategy

Frontier model defaults are designed to make a model broadly usable, not to make an application resilient against prompt injection, data leakage, tool abuse, or policy bypass. That matters because many teams assume a strong base model equals a strong control layer, when the real security boundary is usually the application wrapper, the data pathways, and the permissions granted around the model. Guidance on control design in NIST SP 800-53 Rev 5 Security and Privacy Controls is useful here because it reinforces that security comes from layered controls, not from one setting in one system. In practice, many security teams discover the gap only after users have already trained the application into unsafe behaviour.

How to Tell the Control Layer Is Too Thin

The clearest sign is when your security posture depends on the model “doing the right thing” instead of on explicit application controls. If the only protections are a system prompt, a moderation toggle, or vendor-side defaults, then the organisation is assuming the model will consistently resist adversarial input without independent verification. That is especially weak when the application can read sensitive context, call tools, or act on behalf of a user, because the security problem is no longer just generation quality.

In practice, a thin control layer shows up as a few recurring patterns:

There is no separate test plan for prompt injection, jailbreak attempts, or malicious instruction escalation.
Tool access is broad, static, or not tied to the minimum task requirement.
Confidential data is present in prompts or retrieval context without strong filtering or scoping.
Safety expectations are informal, but failure conditions are not measured or logged.
Developers treat model behaviour as if it were a stable policy engine rather than a probabilistic component.

Anthropic’s Project Glasswing is a useful reference point for thinking about how AI systems need layered protections around model behaviour rather than blind trust in default responses. The guidance breaks down when an application’s control boundaries are unclear, because then any user input, retrieved document, or connected tool can become part of the attack surface.

Common Patterns, Edge Cases, and False Confidence Traps

Hardening an AI system often increases latency, engineering overhead, and operational friction, so organisations have to balance usability against assurance. That tradeoff becomes visible when teams claim the system is “safe enough” because frontier model refusals look good in demonstrations, but they have not tested how those same controls behave under adversarial prompting, context poisoning, or chained tool requests.

Some edge cases are easy to misread. A system may appear well controlled if it refuses obvious harmful prompts, yet still fail when an attacker hides instructions inside retrieved content or when a benign-looking request triggers an unsafe downstream action. Another common misunderstanding is to equate vendor safety settings with application security. Those settings can reduce obvious misuse, but they do not replace access scoping, input handling, output review, or separate monitoring for sensitive actions.

Where agentic workflows are involved, the risk grows further because the model is not only producing text but may also execute actions. In those cases, the right question is not whether the default model behaves politely, but whether the surrounding system can prevent one compromised prompt path from turning into a broader operational or data-access failure. CSA’s agentic AI threat modeling framework is relevant when those tool-using behaviours are central to the application. The warning sign that teams often underestimate is that the weakest control is usually not the model itself, but the unreviewed trust placed around it.

Risk and Threat Considerations

The material risk is control collapse through overtrust in default behaviour. When frontier model defaults are treated as sufficient protection, organisations may expose sensitive data, permit unauthorised tool execution, or fail to detect adversarial prompting that changes the model’s effective behaviour.

Failure mechanism: The weakness usually appears when prompt boundaries, retrieval inputs, and tool permissions are not independently constrained. An attacker or careless user can then steer the model through indirect instruction, context injection, or action chaining, while the surrounding system lacks validation, logging, or least-privilege enforcement.

Impact: The practical result can be leakage of confidential data, unauthorised actions taken through connected tools, policy bypass, and a false sense of safety that hides the problem until the application is already in production.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

MITRE ATLAS address the attack surface, NIST AI RMF and CIS Controls v8 set the technical controls, and ISO/IEC 42001:2023 define the regulatory obligations.

Framework	Control / Reference	Relevance
NIST AI RMF	GOV — Govern	Default-dependent AI security is a governance failure around model risk and oversight.
	MAP — Map	You must map data, tools, and failure modes beyond the model default behaviour.
	MEASURE — Measure	The question is about whether control effectiveness is actually being tested.
Recommendation — Define governance and review gates for AI controls before allowing production use. Map sensitive data flows, tool access, and abuse paths before deploying the system. Measure resistance to prompt injection, leakage, and unsafe action under adversarial tests.
MITRE ATLAS	AML.TA0002 — Reconnaissance	Attackers probe model behaviour to find weak default guardrails and bypass paths.
	AML.TA0005 — Evasion	Prompt injection and indirect instruction are evasion patterns against AI controls.
	AML.TA0009 — Impact	Unsafe defaults can produce leakage, policy bypass, or harmful downstream actions.
Recommendation — Hunt for probing and bypass attempts that reveal weak model-default protections. Test and monitor for evasive prompting that alters model behaviour or policy compliance. Contain and log unsafe outputs so adversarial success does not become operational impact.
ISO/IEC 42001:2023	A.6.2 — AI risk treatment	Overreliance on defaults indicates weak AI risk treatment and assurance.
Recommendation — Establish AI risk treatments that require explicit control validation, not inherited defaults.
CIS Controls v8	8.1 — Establish and Maintain an Asset Inventory	AI systems and their data/tool dependencies must be inventoried to govern exposure.
	6.3 — Require MFA for Externally Exposed Applications	Where AI apps expose access paths, strong access control is part of the protection layer.
Recommendation — Inventory AI applications, connected data sources, and tool integrations before assigning protections. Apply strong access controls to the interfaces that let users reach AI capabilities and tools.

Practitioner Guidance

What to prioritise: Test the full application path, not just the base model. If the model can see private context or invoke tools, validate those boundaries under adversarial conditions before trusting any default safeguard.

What to verify: Confirm that the system has separate controls for input filtering, tool permissioning, sensitive-data handling, and logging of unsafe attempts. A strong refusal rate on simple prompts is not evidence that those controls exist.

Common mistake: Treating the vendor’s safety layer as a substitute for your own governance. The moment the application handles confidential information or actions with real effects, default settings become only one dependency, not the control model.

Practitioner takeaway: If your AI security story depends mainly on the frontier model behaving well by default, you do not yet have a control strategy, only a hope that the model will act as one.