Why do fraud teams need to plan for AI both as a defensive tool and an attacker enabler?

Why This Matters for Security Teams

Fraud teams are no longer evaluating AI as a single capability. They have to assess it as a force multiplier on both sides of the threat model: faster anomaly detection, better case triage, and improved scoring for defenders, but also cheaper phishing, synthetic identity generation, and impersonation at scale for attackers. That dual-use reality is already visible in LLMjacking: How Attackers Hijack AI Using Compromised NHIs and the broader pattern described in the State of Secrets in AppSec, where exposed credentials and weak secret hygiene create immediate abuse windows.

For fraud programs, the mistake is treating AI as a pure efficiency layer. If it is used to accelerate decisions without guardrails, it can amplify false positives, automate analyst overload, and increase the blast radius of compromised identities and secrets. If it is ignored on the attacker side, teams miss the speed and personalization gains that now drive modern fraud campaigns. Current guidance suggests that fraud controls must cover both model usage and model abuse, with clear human escalation paths for high-risk decisions. In practice, many fraud teams discover this only after AI-assisted scams have already inflated case volume and exposed control gaps.

How It Works in Practice

Operationally, the question is not whether AI is useful, but where it is safe to let AI influence fraud outcomes. Defensive use cases usually work best when AI is constrained to recommendation rather than final action: clustering alerts, ranking suspicious sessions, enriching device or payment signals, and surfacing case narratives for analysts. That approach aligns with the control emphasis in OWASP NHI Top 10 and with external threat mapping such as the MITRE ATT&CK Enterprise Matrix, which helps teams think in terms of adversary behaviour rather than isolated events.

The attacker side needs equal attention. Fraud teams should assume AI can be used to generate convincing lures, iterate on social engineering, and adapt to weak detection rules faster than traditional manual fraud rings. That means policy needs to cover both direct model access and indirect AI-assisted workflows. Where AI systems touch sensitive decisions, the safest pattern is tiered review: low-risk actions can be automated, while payment holds, account recovery, identity reproofing, and beneficiary changes retain human approval. Monitoring should look for abnormal prompt volume, unusual API use, rapid changes in fraud score distribution, and repeated attempts to evade step-up checks.

• Use AI to rank and enrich suspicious activity, not to close high-impact cases without oversight.
• Apply strong review thresholds to account takeover, refund abuse, synthetic identity, and payment diversion cases.
• Track model drift, prompt abuse, and sudden changes in fraud pattern shape as control failures, not just model issues.
• Align detection content with threat intelligence from CISA cyber threat advisories and the MITRE ATLAS adversarial AI threat matrix.

Teams that link AI governance, fraud operations, and identity security get better precision and faster response. These controls tend to break down when AI outputs are wired directly into decision engines for high-value transactions because attackers can quickly learn the thresholds and adapt their abuse patterns.

Common Variations and Edge Cases

Tighter AI controls often increase review burden and operational latency, requiring organisations to balance fraud-loss reduction against customer friction and analyst capacity. That tradeoff matters most in environments with high transaction velocity, regulated recovery flows, or large volumes of first-party fraud, where false positives are already expensive. Best practice is evolving, but current guidance suggests that automated decisions should become more conservative as financial impact and reversibility decrease.

There is also no universal standard for when an AI model becomes too risky for direct decisioning. Some teams allow AI to triage low-value events while forbidding it from authorising refunds, credential resets, or policy exceptions. Others use AI only for investigator assist, keeping the decision outside the model entirely. The right boundary depends on your tolerance for error, your appeal process, and whether the model can be monitored and explained well enough to support challenge and audit.

Fraud teams should also account for the attacker’s use of AI outside the transaction flow itself, especially synthetic identities, voice deepfakes, and highly targeted phishing. The strongest programmes pair model governance with identity controls, exception review, and continuous tuning of fraud rules. That approach is reinforced by NHIMG research in the Ultimate Guide to NHIs — Key Challenges and Risks and by the 52 NHI Breaches Analysis, both of which show how compromised digital identities can rapidly turn into repeated abuse.

Related resources from NHI Mgmt Group

• How should security teams reduce the risk of AI tool poisoning?
• How should security teams handle tool discovery for AI agents in MCP environments?
• How should security teams govern tool registration in AI platforms?
• How can security teams prove defensive ROI from AI governance?