AI romance scams are scaling past human detection

At a glance

What this is: This is an analysis of how AI-powered romance scams and deepfake personas now sustain long-running fraud conversations, with the key finding that human instinct is no longer a reliable defense.

Why it matters: It matters because identity teams now have to account for adversarial AI in consumer, workforce, and partner-facing verification flows where trust can be built over time, not just at login.

👉 Read Incode’s discussion of AI-powered romance scams and deepfake identity risk

Context

AI-powered romance scams exploit the gap between what humans think they can verify and what modern generative systems can convincingly imitate. A live face, a matching voice, and natural conversation are no longer enough to establish identity, especially when the conversation can continue for weeks before the fraud attempt surfaces.

For identity and access teams, the core issue is not only fraud prevention at the edge. It is the broader governance problem of proving who or what is on the other end of a digital interaction before trust, money movement, account recovery, or delegated access are allowed to progress.

This is a human identity problem at the surface, but it also reaches into NHI and agentic AI because the same impersonation patterns are increasingly automated, scaled, and operationalised by software systems rather than individual scammers.

Key questions

Q: How should organisations handle romance scams that use deepfakes and AI agents?

A: Organisations should treat them as identity assurance failures, not just user awareness issues. Use stronger verification at high-risk moments, especially for money movement, recovery, and support escalation. The safest approach is to require independent proof before trust is converted into action, because deepfakes can now sustain convincing interactions long enough to bypass instinct.

Q: Why do deepfakes create a bigger identity risk than ordinary phishing?

A: Deepfakes remove the visual and auditory cues people normally rely on to detect fraud. That means an attacker can sustain trust over multiple sessions instead of relying on a single deceptive message. The risk grows when the fake identity is used to request money, access, or recovery actions that appear legitimate.

Q: How can security teams reduce losses from AI-driven impersonation?

A: Focus controls on the moment of decision, not just the moment of first contact. Step-up verification, channel binding, and out-of-band approval can stop a scam after trust has formed but before the attacker gets value. Training helps, but process design matters more because the fraud is now designed to defeat human intuition.

Q: Who should be accountable when AI impersonation leads to fraud losses?

A: Accountability should sit with the teams that own identity proofing, support escalation, and financial approval workflows, because those are the points where trust becomes action. Fraud prevention cannot live only with the end user or only with security. It has to be shared across identity, operations, and finance governance.

Technical breakdown

Deepfake identity verification and why visual trust fails

Deepfakes now defeat the old assumption that a live face and matching voice are sufficient evidence of identity. Once models can sustain an hour-long conversation, the issue is no longer image quality alone but the removal of human telltales as a control point. Verification must move from subjective judgment to evidence-backed identity assurance, especially where financial transfer, account access, or recovery is involved.

Practical implication: treat visual confidence as a weak signal and require stronger verification before any sensitive transaction or identity reset.

AI agents in fraud rings change scale and tempo

The article points to a shift from human-operated scams to AI-operated social engineering, where an agent can generate responses continuously and at near-zero marginal cost. That changes the economics of fraud. Instead of a single attacker spending time on a few targets, a machine can sustain many parallel conversations, adapt wording midstream, and keep pressure on victims until one interaction succeeds.

Practical implication: design controls for volume, concurrency, and automated adaptation, not just for individual scam events.

Continuous identity verification versus one-time proofing

The post contrasts one-time verification with continuous identity verification, which is the more relevant model when trust develops over multiple sessions. In practice, identity assurance should not end after initial onboarding if the interaction can later pivot into money movement, privileged support, or account recovery. The challenge is to bind ongoing behaviour to the verified identity rather than trusting the original proofing event forever.

Practical implication: extend assurance checks into high-risk moments such as transfer requests, recovery flows, and cross-channel escalations.

Threat narrative

Attacker objective: The attacker’s objective is to convert sustained impersonation into money transfer, account control, or another high-value action that appears authorised by the victim.

1. Entry occurs through a convincing AI-generated persona that initiates contact on social or messaging platforms and sustains trust through realistic video and voice interactions. Escalation follows when the attacker uses that trust to request money, credentials, or account actions while shifting channels to avoid detection. Impact is financial loss, compromised accounts, and in some cases organisational disruption when impersonation is used against staff or support workflows.

Breaches seen in the wild

• Cisco DevHub NHI breach — IntelBroker exploited exposed Cisco credentials, API tokens and keys in DevHub.
• DeepSeek breach — DeepSeek breach exposed 1M+ log lines and sensitive secret keys.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Deepfake romance scams are not a better phishing trick, they are an identity assurance failure. The attacker is no longer merely spoofing a message, they are sustaining a believable identity over time across video, voice, and chat. That means the control failure is not just user awareness, it is the absence of a stronger proofing and step-up model at the point where trust turns into money movement or recovery. Practitioners should treat this as an identity governance issue, not a content moderation problem.

Continuous identity verification is becoming the new boundary for high-risk digital trust. One-time proofing assumes the relationship between the verified subject and the transaction remains stable, but AI-driven impersonation breaks that assumption across channels and sessions. The implication is that assurance must be evaluated at the moment of risk, not only at account creation. Teams that govern KYC, support, and account recovery need to align on the same risk trigger.

AI agents are now a fraud scaling layer, not just a scam enabler. The article’s warning about AI agents replacing human operators in fraud rings matters because it changes both tempo and volume. That creates a trust graph problem where every interaction can be chained into the next one at machine speed. Security teams should assume the fraudster’s conversation engine is now autonomous enough to outpace manual review.

Identity fraud is moving from single-point impersonation to session-long behavioural persistence. That shift creates a named governance gap we can call persistent persona trust debt: the longer a fake identity is allowed to remain believable, the more every subsequent control inherits the original trust error. Practitioners should interpret this as a problem of accumulated assurance debt across the interaction lifecycle, not a one-time verification miss.

From our research:

• The average estimated time to remediate a leaked secret is 27 days, despite 75% of organisations expressing strong confidence in their secrets management capabilities, according to The State of Secrets in AppSec.
• Only 44% of developers are reported to follow security best practices for secrets management, according to The State of Secrets in AppSec.
• For the related agentic fraud landscape, LLMjacking: How Attackers Hijack AI Using Compromised NHIs shows how exposed credentials can be weaponised in minutes, not days.

What this signals

Persistent persona trust debt: identity programmes now need to account for the way trust compounds across multiple interactions before a single high-risk action is taken. The practical shift is toward step-up verification, stronger recovery governance, and tighter approval logic wherever a channel can be socially engineered into authority.

The same pressure that has already fragmented secrets management will show up in fraud controls if identity, finance, and support teams work from different risk thresholds. The next failure mode is not simply a convincing fake, it is inconsistent governance across the points where a fake becomes actionable. For identity leaders, that means binding verification policy to transaction risk rather than user convenience.

For practitioners

• Add step-up verification at money movement points Require stronger identity proofing before wire transfers, account recovery, beneficiary changes, or support-triggered privilege changes. Use independent verification channels and do not rely on the same conversation that requested the action.
• Limit trust escalation across channels Do not let a persuasive chat, video call, and phone call count as three separate proofs of the same identity. Require that the verified identity be bound to the interaction context before approvals are granted.
• Train support and finance teams on impersonation patterns Teach teams to treat urgency, secrecy, channel switching, and refusal of in-person or out-of-band verification as common fraud indicators. Pair that training with a clear escalation path when the requested action is unusually sensitive.
• Instrument high-risk verification workflows Log and review when an interaction moves from chat to voice to video and then to financial or recovery requests. These transitions are often where the fraud attempt becomes operationally dangerous.

Key takeaways

• AI-powered romance scams are now an identity assurance problem because deepfakes can sustain trust long enough to trigger financial or access actions.
• The scale risk is rising because AI agents can automate scam conversations, making impersonation cheaper, faster, and harder to contain.
• The practical defense is to move verification to the point of risk, especially for transfers, recovery, and support escalation.

Key terms

• Deepfake Identity Fraud: Deepfake identity fraud uses generated video, audio, or text to impersonate a real person well enough to pass human judgment and trigger action. In practice, the fraud succeeds when the fake identity is believable long enough to extract money, access, or other value before verification catches up.
• Continuous Identity Verification: Continuous identity verification is the practice of rechecking trust at the moments that matter, not only at onboarding. It matters because identity can drift across sessions, channels, and contexts, so high-risk actions need stronger proof than the original login or account creation event.
• Persistent Persona Trust Debt: Persistent persona trust debt is the accumulation of misplaced confidence in an identity that remains believable over time. The longer a fake persona goes unchallenged, the more later approvals, transfers, and escalations inherit the original trust failure and amplify the loss.

What's in the full article

Incode's full post covers the operational detail this post intentionally leaves for the source:

• The transcript of Ricardo Amper’s commentary on deepfake-driven romance scams and why traditional detection fails.
• The real-world examples of impersonation attempts against finance and support workflows, including the treasury call incident.
• The concrete verification practices discussed for platforms, social networks, and consumer identity flows.
• The longer discussion of AI agents scaling fraud operations and what that means for future verification design.

👉 The full Incode post includes the transcript, examples, and verification guidance discussed in the episode.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an identity security programme, it is worth exploring.