AI-generated fraud pressure is exposing trust and safety org design gaps

At a glance

What this is: Sift argues that AI-driven fraud is pushing organisations beyond transaction-only fraud operations and toward trust and safety structures built for account abuse, social engineering, and synthetic identities.

Why it matters: IAM, fraud, and identity teams need this shift because the control problem is no longer limited to stopping bad payments; it now includes governing account trust, human decision points, and cross-functional escalation paths.

By the numbers:

• Transaction volume on the Sift Global Data Network grew 18% in 2025.
• Payment fraud attempt rates held steady at around 3.25%.

Context

AI has changed fraud from a mostly transaction-level detection problem into a broader trust governance problem. When abuse shifts from obvious payment attacks to account creation fraud, account takeover, social engineering, and synthetic identity abuse, a narrow fraud queue is no longer enough to manage the risk surface.

The article’s central point is organisational, not purely technical: the operating model must match the abuse profile. That matters to IAM and identity verification teams because account trust, step-up decisions, and review escalation now sit on the same risk path as traditional fraud controls.

Key questions

Q: How should organisations structure fraud and trust and safety teams when abuse spans accounts and transactions?

A: Use the abuse surface as the organising principle. If the problem is only payment fraud, a focused fraud team may work. If the platform faces account takeover, account creation abuse, policy abuse, or synthetic identities, a trust and safety model with shared escalation and cross-functional ownership is more realistic.

Q: Why do loss-rate metrics fail to show whether fraud controls are working?

A: Loss rate only captures what was stopped, not what was wrongly blocked, delayed, or pushed into a better attacker workflow. Teams need approval rate, false positive rate, customer churn after fraud events, and review throughput to understand the real operational and business effect of controls.

Q: What breaks when a fraud team is built only around payment review?

A: It misses abuse that does not look like a payment event, including account takeover, synthetic identities, and human-targeted social engineering. The result is fragmented ownership, weak escalation, and controls that optimise one narrow queue while trust degrades elsewhere.

Q: How should fraud leaders explain trust and safety investment to executive teams?

A: Frame trust as infrastructure, not overhead. Executives usually understand revenue, customer retention, and operational efficiency more quickly than fraud jargon, so the strongest case is that better trust governance preserves legitimate transactions, reduces churn, and limits customer harm.

Technical breakdown

Why AI scales abuse faster than manual fraud operations

Fraud teams used to rely on pattern recognition, slow feedback loops, and human review to keep pace with abuse. AI compresses attacker effort, increases message quality, and lowers the cost of experimentation, which means fraud-as-a-service can adapt faster than a manual queue can learn. The important change is not that fraud exists in new places, but that the same abuse can be generated at a speed and volume that make reactive operations lag by design. That pushes programmes toward higher-fidelity signals, better triage, and tighter linkage between identity events and business risk.

Practical implication: treat AI-amplified abuse as a throughput problem as much as a detection problem.

Trust and safety org design versus traditional fraud teams

A traditional fraud team usually centres on discrete payment decisions, measurable loss reduction, and tight operational controls. A trust and safety organisation is broader: it has to handle account creation abuse, account takeover, policy abuse, content scams, and cross-functional escalation because the abuse surface is tied to platform trust, not just transactions. That changes how cases are routed, how success is measured, and who owns remediation. In identity-heavy environments, this also changes where IAM, verification, and fraud controls intersect, because account trust becomes a programme-level concern rather than a single-team metric.

Practical implication: align team structure to the full abuse surface before trying to tune individual detection models.

Why metrics need to move beyond loss rate

Loss rate is useful but incomplete because it measures what was stopped, not what was enabled or incorrectly blocked. Modern fraud operations need outcome metrics that include approval rates, churn after fraud events, customer false positives, and operational efficiency in review and response. Those metrics show whether controls are protecting the business or simply shifting pain from fraud losses to legitimate customers. For IAM and trust governance, the lesson is that access and abuse decisions must be judged by downstream business impact, not just detection precision.

Practical implication: pair fraud-loss reporting with customer-impact and approval metrics to see the real control trade-off.

Threat narrative

Attacker objective: The objective is to scale profitable abuse while blending into ordinary account and trust workflows.

1. Entry begins with AI-assisted social engineering, synthetic identity creation, or account abuse that reaches the platform through normal onboarding and interaction paths.
2. Credential or account access is then abused through account takeover, making legitimate-looking identities the vehicle for fraud rather than isolated payment attacks.
3. The attacker scales abuse across transactions, content, or internal trust channels, using human-like behaviour to evade narrow fraud checks.
4. Impact is organisational: higher review burden, weaker trust signals, and pressure on teams whose operating model was built for slower, simpler fraud patterns.

Breaches seen in the wild

• Gemini AI Breach — Google Calendar Prompt Injection: Gemini AI assistant prompt injection attack leaks sensitive Google Calendar data.
• Gemini CLI prompt injection flaw 2025: Tracebit showed a poisoned README could make Gemini CLI run hidden commands and exfiltrate developer secrets; Google fixed it in 0.1.14.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI-driven fraud is now an identity governance problem as much as a detection problem. When attackers use synthetic identities, account takeover, and human-targeted manipulation, the control boundary moves upstream into verification, trust scoring, and escalation design. That means fraud, IAM, and identity verification can no longer operate as separate optimisation tracks. Practitioners should treat trust decisions as governed identity events, not just fraud cases.

Trust and safety organisation design is the real control surface. The article is right that team structure determines whether a programme can absorb multi-vector abuse. A payment-only fraud queue cannot govern platform trust when abuse includes accounts, people, and policy misuse. Practitioners should map ownership by abuse type, then align escalation authority and review capacity to that map.

Outcome metrics matter more than loss metrics once abuse becomes synthetic. Loss rate alone hides false positives, customer attrition, and the cost of slow escalation. This is where trust and safety programmes either become business infrastructure or remain a cost centre. Practitioners should redefine success around approval quality, customer harm, and response latency, not just blocked fraud.

Named concept: trust infrastructure drift. This is the gap between how a company thinks trust is governed and how abuse actually moves through its products, identity flows, and human review paths. As AI lowers the cost of creating convincing abuse, drift grows unless organisations continuously realign ownership, metrics, and escalation. Practitioners should audit whether their operating model still matches the current abuse surface.

From our research library:

• Nearly 60% of companies reported that fraud losses were still increasing in 2025.
• Read next: Zero Trust for AI Agents

What this signals

Trust infrastructure drift: most fraud organisations do not fail because they lack models, but because their operating structure no longer matches how abuse now moves through accounts, people, and products. When AI reduces attacker friction, the governance question becomes whether the programme can still route, prioritise, and escalate the right events fast enough.

For IAM and identity verification teams, the practical shift is to treat account trust as a governed lifecycle rather than a one-time onboarding decision. That means escalation paths, review thresholds, and assurance checks must be designed around changing abuse patterns, not just static policy.

For practitioners

• Map your abuse surface by type Separate payment fraud, account takeover, account creation abuse, policy abuse, and synthetic identity risk before choosing team structure or tooling.
• Redesign escalation around trust events Route account trust failures, suspicious onboarding, and high-risk human interactions through defined escalation paths instead of leaving them inside a payment-only queue.
• Measure business outcomes alongside loss Track approval rate, false positives, churn after fraud events, and review throughput so leadership can see the cost of control decisions.
• Hire for investigative curiosity Prioritise analysts who can pursue anomalies, question assumptions, and adapt as fraud patterns change rather than relying only on credentials or domain pedigree.

Key takeaways

• AI-driven abuse is forcing fraud teams to move beyond payment-centric operations and into broader trust and safety governance.
• The article’s core evidence is operational, not theoretical: transaction growth and stable payment fraud attempts point to attackers shifting toward accounts, identities, and human manipulation.
• The decisive control gap is organisational fit, because team structure, metrics, and escalation paths must match the full abuse surface if the programme is going to scale.

Key terms

• Trust And Safety: Trust and safety is the combined discipline of preventing abuse, reducing harm, and preserving legitimate participation in a digital community. In identity programmes, it links verification, moderation, and lifecycle governance so account confidence and user experience are managed together.
• Account Takeover: Account takeover is unauthorized use of a legitimate account after an attacker obtains valid access through stolen credentials, tokens, or trusted integrations. The key security problem is that the resulting activity often looks normal to logs and controls, which makes containment and attribution harder than in a forced-entry breach.
• Synthetic Identity: A synthetic identity is a software-based actor that can authenticate, request access, and execute actions without being a human user. In practice, this includes AI agents, bots, service accounts, tokens, and other machine identities that need clear ownership, scope, and revocation.
• False Positive: A false positive is a scanner result that looks like a secret but is not actually sensitive. In secret governance, false positives matter because they consume analyst time, weaken trust in alerts, and can delay response to the findings that truly change exposure and access risk.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and IAM concepts that support stronger identity control decisions. It is useful for practitioners who need to connect identity governance to broader security and operational risk programmes.