Notifications
Clear all
Topic starter
10/07/2026 9:23 pm
TL;DR: Open-weight models with stripped refusal behavior can now generate convincing phishing, deepfake lures, and other fraud content at low cost, while reported AI-related fraud losses reached $893 million in the FBI’s 2025 Internet Crime Report and researchers found 100,000+ exposed AI servers, according to Incode. Static identity verification and content-based trust checks are no longer sufficient when authenticity can be synthesized on demand.
NHIMG editorial — based on content published by Incode: What Every Fraud Leader Should Know About Locally Run AI
By the numbers:
- IBM X-Force found that an AI assistant could produce a convincing phishing email in roughly 5 minutes, versus the 16 hours an experienced human operator typically needs.
Questions worth separating out
Q: How should security teams handle identity verification when AI can synthesize voices and messages?
A: Security teams should stop relying on content quality as proof of identity and move to layered verification.
Q: Why do open-weight AI models increase fraud and impersonation risk?
A: Open-weight models increase risk because the operator can modify safety behaviour locally and use the model to generate convincing fraud content without provider-side policy enforcement or logging.
Q: What do security teams get wrong about deepfakes and phishing detection?
A: Many teams still look for obvious defects such as poor grammar, awkward phrasing, or low-quality audio.
Practitioner guidance
- Replace content trust with transaction trust Require out-of-band confirmation, device-binding, and step-up controls for payments, account recovery, and privileged requests.
- Add synthetic-media resistance to fraud workflows Test identity journeys against voice cloning, deepfake video, and AI-written phishing.
- Reclassify local model hosting as a governance risk Inventory where open-weight models run, who can modify them, and whether any guardrail removal or jailbreak tooling is reachable on those systems.
What's in the full article
Incode's full article covers the operational detail this post intentionally leaves for the source:
- The specific local-model manipulation steps used to strip refusal behaviour from an open-weight model.
- The full set of fraud scenarios the author tested, including phishing optimisation and impersonation refinement.
- The article’s discussion of AI-related fraud losses, deepfake examples, and the shift away from content-based trust.
- The author’s recommended fraud and verification controls for high-risk business workflows.
👉 Read Incode's analysis of locally run AI and fraud verification risk →
Open-weight AI and fraud verification: what breaks in identity trust?
Explore further
10/07/2026 9:46 pm
Authenticity is no longer a reliable identity control when the content itself can be synthesized. The article shows that AI can generate persuasive phishing, impersonation, and executive-style communication at low cost and high speed. That breaks the assumption that a message, call, or video carries enough trust signal to support high-value decisions. The implication is that fraud and identity programmes must stop treating generated content as an edge case and start treating it as a baseline condition.
A few things that frame the scale:
- The FBI’s 2025 Internet Crime Report documented $893 million in AI-related fraud losses, according to LLMjacking: How Attackers Hijack AI Using Compromised NHIs.
- The same research notes that when AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes, and as quickly as 9 minutes in some cases.
A question worth separating out:
Q: Who is accountable when synthetic identity fraud succeeds in a business process?
A: Accountability should sit with the process owner who accepted a weak trust signal, the control owner who allowed it, and the identity team that failed to require stronger verification for the risk level. Frameworks such as NIST CSF and NIST SP 800-53 become relevant because they tie governance, access control, and monitoring to business-impacting decisions.
👉 Read our full editorial: Open-weight AI makes fraud verification assumptions break down
