AI romance scams are scaling past human detection

At a glance

What this is: This is an analysis of AI-enabled romance scams and deepfakes, showing that convincing live impersonation can now outlast human detection and drive financial fraud.

Why it matters: It matters because identity teams, fraud teams, and platform owners can no longer rely on visual or audio cues alone, and verification has to move into the trust flow before relationships and transactions are established.

Context

AI romance scams are a human identity problem that now behaves like a scalable fraud operation. Incode describes how deepfakes and AI agents can hold convincing conversations over video, chat, and voice long enough to build trust before requesting money, which means the old assumption that people can simply spot the fake is no longer dependable.

The governance gap is not only in consumer vigilance but in the systems that establish trust. When a platform accepts a face, a voice, or a scripted conversation as proof of identity, it creates an opening for impersonation, account takeover, and payment fraud that is difficult to unwind after trust has already formed.

Key questions

Q: What breaks when romance scams are driven by deepfakes instead of real people?

A: The normal human detection model breaks first. Visual and audio cues stop being reliable, so users can no longer safely judge authenticity from a call, a voice note, or a video clip alone. The practical response is to move from instinct-based trust to verified identity checks before the interaction turns into money movement or account access.

Q: Why do AI-powered romance scams create higher fraud risk than traditional phishing?

A: They sustain trust over time instead of forcing a fast decision. A synthetic persona can keep a conversation going for weeks or months, adapt to the victim’s responses, and wait for the right moment to ask for money. That persistence increases both conversion rates and the eventual loss when the fraud succeeds.

Q: How can teams tell when deepfake-based impersonation is bypassing normal controls?

A: Look for repeated channel switching, urgency around money, reluctance to use standard verification paths, and relationships that intensify faster than normal. Those are not proof on their own, but they are strong indicators that the interaction is being engineered to defeat the user’s confidence and the organisation’s review process.

Q: What should organisations do when customer conversations may involve synthetic identity?

A: Put verification ahead of trust, not after it. Organisations should require verified profiles, step-up proofing, and transaction controls before a conversation can escalate into payment, access, or sensitive personal disclosure. That reduces the chance that a well-run impersonation campaign can convert attention into loss.

Technical breakdown

Why live deepfakes defeat human detection

Deepfakes used to fail because lip sync, voice timing, and motion artifacts exposed them within seconds. The article shows that those tells have largely disappeared, and attackers can now sustain a credible live video conversation for an hour or more. That changes the problem from spotting obvious fakes to distinguishing synthetic from authentic identity signals in real time. For identity security, this is a verification challenge, not a perception challenge.

Practical implication: treat human observation as a weak signal and move verification earlier in the trust flow.

How AI agents scale romance fraud

The article distinguishes between a simple deepfake clip and an AI agent that can carry the conversation forward. In a fraud ring, the agent can respond naturally, maintain context, and continue the deception with almost no marginal cost, which removes the human labor constraint that used to limit scam volume. That makes romance fraud more industrial, more persistent, and harder to triage by normal customer-support or fraud-review workflows.

Practical implication: assume fraud operations can now sustain long-running, adaptive interactions without human operators in the loop.

Why identity verification must replace instinct

The core control shift is from subjective trust cues to verifiable identity proofing. The article points to continuous identity verification, verified profiles, and device or platform-level checks as the mechanisms that can establish who a person is before high-trust interactions continue. In practice, this is about moving trust decisions into an enforceable identity layer rather than leaving them to the user’s judgment after the relationship is already established.

Practical implication: embed identity verification before money movement, privileged contact, or extended relationship-building can occur.

Threat narrative

Attacker objective: The attacker aims to convert synthetic trust into money, account access, or both, while keeping the victim engaged long enough to bypass suspicion.

1. Entry begins when the attacker establishes contact through a dating platform, social network, or messaging channel using a synthetic persona that appears authentic.
2. Credential access is replaced by trust capture: the attacker uses video, voice, and conversational persistence to win confidence, then steers the victim toward money transfer or account actions.
3. Escalation occurs as the scam moves from casual messaging to repeated video calls, stronger emotional dependency, and higher-value requests that reduce skepticism.
4. Impact is financial loss and emotional harm, with victims wiring funds or sharing access based on a relationship that was never real.

Breaches seen in the wild

• Arup deepfake fraud 2024: Deepfakes of Arup's CFO and colleagues on a video call led a Hong Kong employee to transfer HK$200 million (about US$25.6m) to fraudsters.
• CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Human instinct is no longer an identity control: This article shows that visual and vocal cues can be manufactured well enough to sustain trust for hours or months. That means fraud resistance cannot depend on a person noticing discomfort, hesitation, or a clipped voice. The implication is that identity assurance has to move from perception to verification before trust is granted.

AI romance fraud is becoming a machine-scaled trust attack: The limiting factor is no longer how many humans a fraud ring can staff. AI agents can maintain relationship context, vary responses, and keep pressure on victims at near-zero marginal cost, which turns romance scams into a scalable identity abuse problem rather than a one-off social-engineering trick. Practitioners need to treat this as an operational fraud model, not just a user-awareness issue.

Verified identity must become part of the relationship layer: Platforms that allow prolonged high-trust interactions without continuous identity proofing are effectively asking users to self-police synthetic personas. That assumption fails when deepfakes can pass as live people and when AI agents can carry the conversation after initial contact. The implication is that identity verification has to be woven into onboarding, escalation, and payment flows.

Deepfake detection and identity proofing are converging controls: The article points to models that detect pixel-level inconsistencies invisible to humans, which means fraud defense is shifting toward machine-assisted verification. This creates a named concept worth tracking: trust inversion at the point of contact: the burden moves from the user deciding whether someone is real to the system proving it before trust can deepen. Practitioners should design for that inversion.

The fraud stack now spans consumers and enterprises: The same impersonation mechanics used in romance scams can also target treasury teams, executives, and service channels. That cross-context reuse matters because controls built only for consumer safety or only for internal fraud miss the shared identity pattern. The practical conclusion is to align consumer-facing verification, workforce trust controls, and transaction approval rules around the same impersonation risk.

From our research library:

• 70% of organisations grant AI systems more access than they would give a human employee performing the exact same job, according to the 2026 Infrastructure Identity Survey.
• Systems with least-privileged AI access had a 17% incident rate vs 76% for over-privileged systems. Organisations failing to scope AI access properly are 4.5x more likely to experience a security incident, according to the 2026 Infrastructure Identity Survey.
• Read next: Zero Trust for AI Agents

What this signals

Trust inversion at the point of contact: Identity teams should assume that the first convincing interaction may already be synthetic, so the control objective shifts from detection after engagement to proof before escalation. That is especially important where a relationship can mature into payment, access, or account recovery without any formal review checkpoint.

The programme implication is broader than romance scams. The same impersonation pattern can target finance owners, customer-support channels, and executive assistants, which means verified identity and step-up checks need to sit in the workflow, not in a training deck.

AI access is being over-granted in adjacent research, with 53% of security leaders expecting AI to run major portions of infrastructure autonomously within three years according to the 2026 Infrastructure Identity Survey. The lesson for identity practitioners is that synthetic actors will keep finding the weakest trust boundary unless verification is built into the boundary itself.

For practitioners

• Require verified identity before trust deepens Use verified profiles, IDV, or equivalent proofing before extended messaging, private video, or payment requests are allowed to continue.
• Instrument continuous verification in high-trust flows Add repeatable checks at the points where a conversation turns into a financial request, account recovery, or sensitive escalation.
• Train fraud teams to treat deepfake cues as insufficient Update playbooks so reviewers look for relationship timing, payment pressure, and channel switching, not just visual anomalies.
• Separate emotional engagement from transaction approval Require independent approval paths for wire transfers, gift payments, or account actions that originate in a personal conversation.

Key takeaways

• Deepfake-enabled romance scams turn human trust cues into a liability, because the attacker can now hold a convincing interaction long enough to solicit money or access.
• The article shows that the old telltales are disappearing and that AI agents can scale impersonation without the labour limits that constrained earlier fraud rings.
• The practical defence is to move trust decisions into verified identity controls before relationship-building, payment, or privileged contact can continue.

Key terms

• Deepfake: Synthetic or altered media created with AI or machine learning so that a person appears to say or do something they never did. In security terms, deepfakes are trust attacks that can distort identity verification, approval workflows, and fraud detection.
• Identity verification: Identity verification is the process of confirming that a user, workload, or agent is the entity it claims to be before access is granted. In AI-heavy environments, that verification must include the requester, the system acting on its behalf, and the sensitivity of the action.
• Romance Fraud: Romance fraud is a scam in which an attacker creates a false online identity to build trust with a target and then exploit that relationship for money or personal data. It combines emotional manipulation with deception, and often develops over time before the fraud becomes visible.
• Continuous Verification: A Zero Trust practice that re-evaluates trust during the session instead of relying on a single successful login. The control is stronger when context signals are available in real time and when the identity programme can act on those signals without creating excessive exceptions.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.