Threat model mismatch is the real failure in application security

At a glance

What this is: This article argues that security controls fail when they model a passive user instead of an attacker who controls the runtime or automation pipeline.

Why it matters: For IAM, NHI, and security teams, the lesson is that access, trust, and execution assumptions must be tested against real attacker capabilities, not policy intent alone.

👉 Read Arxan Technologies' analysis of threat model failure in application security

Context

Security controls often break at the boundary between the attacker a system expects and the attacker it actually faces. In this case, the article uses two examples, a spoofed recruiting email and IEEE 1735, to show that threat model mismatch is the root problem in many application security failures. The primary lesson is not about one vendor or one cryptographic scheme, but about how assumptions about human review, execution trust, and runtime control can be exploited.

That matters directly to identity and access governance because the same pattern appears whenever a system trusts a human, a tool, or a workload more than it should. In NHI and agentic AI programmes, the equivalent mistake is assuming a credential, an agent, or a tool invocation will behave within the limits designers imagined. When that assumption collapses, controls that look strong on paper can still fail in production.

Key questions

Q: What breaks when application security controls assume a passive attacker?

A: Controls break when they are designed for a user who only reads protected content, but the attacker can probe the runtime, observe errors, or automate trust abuse. In that case, encryption, identity cues, and usability choices can become part of the attack path rather than the defence. The fix is to validate assumptions against active adversaries, not ideal users.

Q: Why do personalised AI-generated lures increase security risk?

A: They reduce the signal gap between legitimate outreach and malicious outreach by using public data to create highly contextual messages at scale. That makes traditional user intuition less reliable and raises the burden on mail security, identity verification, and anomaly detection. The risk is not just better phishing copy. It is automated trust exploitation.

Q: Where do runtime protections fail in hostile execution environments?

A: They fail when the control protects data in storage but not the process that handles it. If an attacker can instrument the environment, craft inputs, or read outputs repeatedly, the software may leak keys, logic, or protected content through its own behaviour. Runtime integrity checks matter because they constrain what a compromised environment can learn or alter.

Q: Should security teams treat threat modelling as a governance control?

A: Yes. Threat modelling should be treated as a governance control because it determines whether every other control is aimed at the real attacker. If the model is wrong, compliance can still look good while the system remains vulnerable. Teams should require evidence that assumptions were tested against automation, runtime access, and adversarial behaviour.

Technical breakdown

Threat model mismatch in application security

A threat model defines what an attacker can do, what they cannot do, and which environment boundaries are trusted. In the IEEE 1735 example, the standard assumed a passive reader of protected files, but the real attacker could control the execution environment and observe error behaviour. That shift turns safe-looking encryption into an oracle problem. The cryptography was not the failure; the security boundary was. Once the tool itself becomes part of the attack surface, design choices like verbose errors, unauthenticated ciphertext handling, and trusted runtime assumptions become exploitable.

Practical implication: validate every protection against the strongest realistic attacker, not the user path you intended.

Why AI-driven social engineering scales the same failure pattern

The recruiting-email example shows how AI changes the economics of trust exploitation. Public profile data can be scraped, classified, and turned into personalised lures at machine speed, which means the attack no longer depends on generic phishing quality. The attack surface is the personalised trust signal itself. The system was designed for human outreach, but the attacker used automation and generative text to imitate legitimate intent. That is a security design failure, not just a messaging trick.

Practical implication: assume public identity data can be operationalised into automated trust abuse at scale.

Runtime integrity matters when the attacker is already inside

When the adversary can interact with the running environment, static cryptography alone is insufficient. White-box cryptography and runtime integrity checks exist because keys and logic are exposed to inspection, tampering, and instrumentation in hostile execution contexts. The important point is not the specific technique, but the architectural shift: protections must survive active probing, not just storage confidentiality. That same principle now applies to software agents, distributed tooling, and other systems that execute under partially trusted conditions.

Practical implication: pair secrecy controls with runtime integrity, because confidentiality without execution control leaves a gap.

Threat narrative

Attacker objective: The attacker aims to convert trust into access, whether that means credential capture, protected-information exposure, or control of the execution boundary.

1. Entry occurs through personalised recruitment lures built from scraped public identity data and delivered through spoofed infrastructure that looks legitimate at first glance.
2. Escalation happens when the target trusts the message because the attacker has automated the generation of context-aware, high-confidence social engineering at scale.
3. Impact is achieved when trust is extracted from the recipient or when the protected execution environment is manipulated through oracle-style behaviour, exposing secrets or protected logic.

NHI Mgmt Group analysis

Threat model mismatch is the hidden control failure behind many security incidents: the strongest-looking protection can fail if it was built for a passive user and the attacker is active, instrumented, and automated. IEEE 1735 is a clean example because the cryptography was not the problem. The control gap was the assumption about who could interact with the system and how. Practitioners should treat the threat model itself as a security control, not a design appendix.

OWASP Agentic Applications Top 10 belongs in the same conversation as application security here: once AI-assisted actors can generate, personalise, and iterate trust abuse at machine speed, the boundary between application risk and identity abuse collapses. That is especially relevant for IAM and NHI teams because machine-speed interaction changes what counts as a realistic attacker capability. Practitioners should evaluate controls against automation, not only against human misuse.

DeepSeek breach shows how exposed secrets and exposed systems create compounding risk: once sensitive material is reachable by an attacker, tooling that assumes benign use can turn into a disclosure path. The article’s deeper lesson is that runtime exposure, logging behaviour, and trust boundaries must be reviewed together. Practitioners should assume that any exposed execution path can become a secret disclosure path.

Trust signals are now an attack surface, not a comfort signal: personalised identity cues, recruiter language, and domain lookalikes can all be manufactured cheaply. That means identity verification, mail security, and user training must be designed around adversarial automation, not just human error. Practitioners should treat trust synthesis as a control problem, not a communications problem.

Application security programs need a named concept for this failure mode: threat model drift: the control design stays static while attacker capability changes. This is where procurement, architecture review, and red-team assumptions often fall apart. Practitioners should require evidence that the control still holds when the attacker can automate, instrument, and observe the system.

What this signals

Threat model drift is becoming a practical governance issue, not just a design critique. As attackers use AI to scale trust abuse and probe software boundaries, control reviews need to ask whether the expected attacker is still realistic. Teams that still test only for manual misuse will miss how fast automation changes the risk surface.

Identity and access programmes should treat public identity data, inbox routing, and execution trust as linked control layers. When those layers are separable on paper but blended by the attacker in practice, the governance model fails first. That is where mail security, verification, and runtime integrity need to be assessed together, not as isolated workstreams.

For practitioners

• Document the attacker model explicitly Record what the control assumes the attacker cannot do, then test whether that assumption still holds when the attacker can automate, instrument, or control the runtime.
• Review error handling as an attack surface Remove descriptive error output where it can be used as an oracle, and separate internal diagnostics from externally visible responses.
• Test security controls against AI-assisted abuse Run exercises that simulate scraped identity data, personalised lures, and repeated query attempts so teams can see where trust fails under scale.
• Pair encryption with runtime integrity Use runtime checks, tamper detection, and environment attestation where software executes in partially trusted or hostile conditions.
• Reassess trust in public identity data Treat public profile fields, domain cues, and inbox routing as inputs that can be weaponised, and feed that into identity verification and mail security controls.

Key takeaways

• The article shows that security often fails because the control was built for the wrong attacker, not because the cryptography was weak.
• The same trust mismatch appears in AI-assisted social engineering, where public identity data can be converted into scalable deception.
• Security teams should make the threat model testable, then verify that runtime controls still hold when the attacker can automate and instrument the system.

Key terms

• Threat Modeling: A structured way to identify where an application can be entered, abused, or misused before or during development. It maps trust boundaries, access points, and likely abuse cases so teams can design controls that match the way attackers and misuse actually happen.
• Oracle Attack: An oracle attack occurs when a system reveals useful information through its responses, such as error messages, timing, or behavioural differences. The attacker uses repeated probing to infer protected data or logic without directly breaking the cryptography or authentication layer.
• Runtime Integrity: Runtime integrity is the assurance that what a system executes matches what defenders expect at the moment of execution. It matters because malware or exploit code can change memory, cache, or process behaviour without changing the on-disk file, which makes artifact-based checks incomplete on their own.
• Threat Model Drift: Threat model drift is the gap that opens when a threat model no longer matches the application it describes. It happens as code, infrastructure, and data flows change over time, which can make old assumptions unsafe during reviews, triage, or incident response.

What's in the full article

Arxan Technologies' full post covers the operational detail this post intentionally leaves for the source:

• The paper-like walkthrough of IEEE 1735 failure mechanics and why its assumptions broke under active probing.
• The detailed explanation of how AI-generated recruiting lures were assembled from public identity data and routed through spoofed infrastructure.
• The specific cryptographic and runtime integrity discussions around white-box cryptography in hostile execution environments.
• The references and source material that support the article's argument about threat model failure and control design.

👉 Arxan Technologies' full post covers the IEEE 1735 breakdown, AI-assisted social engineering, and runtime trust assumptions in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is designed for practitioners who need to connect identity controls to real-world attack behaviour and lifecycle governance.