Voice AI Agent Security and Fraud Guide: Deepfakes, Vishing and Controls

A voice on the phone is no longer evidence of who is speaking, and the AI voice agents organisations now put on their own phone lines can be talked into acting for the wrong person. Voice cloning lets criminals impersonate executives, family members and officials, and AI callers let them run vishing at scale; at the same time, contact centres and products are deploying voice agents that can read account data, reset credentials and move money. This guide covers both sides: defending people and processes against synthetic voice fraud, and securing the voice AI agents you build or buy, including authentication, tool permissions, observability and evaluation. It is written for fraud, contact-centre, identity and AI platform teams. For wider impersonation across video, email and messaging see the Deepfake and AI Impersonation Guide.

Key takeaways

• Stop treating voice as an authenticator. NIST SP 800-63B-4 states that biometric comparison based on voice shall not be used; caller ID, voice recognition and knowledge questions all fail against cloned voices and breached data.
• Verify through a channel the caller does not control. Call back on a number you already hold, confirm in an authenticated app, or require a second approver; this is the control that survives better cloning.
• A voice agent is an identity with privileges. Give it its own credentials, scope its tools to the minimum, and require the caller to be authenticated before any account-changing or payment action.
• Speech is untrusted input. Prompt injection, social engineering and pressure tactics reach a voice agent through the transcript, so the policy checks that apply to chat agents apply here, before the agent speaks or calls a tool.
• Log and evaluate the conversation, not just the call. Keep transcripts, tool calls, authentication outcomes and handoffs, and test agents regularly with adversarial scripts and synthetic voices.

The two problems: voice AI against you, and your own voice agents

Government advisories describe the inbound threat clearly. The FBI's December 2024 public service announcement (I-120324-PSA) reports criminals using short clips of a person's voice to clone it, fake an emergency and demand money, and using cloned voices to gain access to bank accounts. In May 2025 (I-051525-PSA) the FBI warned that since April 2025 actors had sent text and voice messages, including AI-generated voice messages, impersonating senior US officials to build trust before moving targets to another platform and taking over accounts.

The second problem is newer. A voice agent on your phone line or in your app combines a speech front end, a language model and tool access to your systems. If an attacker can talk it into revealing data, resetting a credential or changing a payee, the agent becomes the impersonation target rather than a human agent. The same help desk social engineering that worked against people, as in the Co-op help desk attack, now has an automated counterpart.

Why traditional phone authentication fails

• Voice biometrics: NIST SP 800-63B-4 (Section 3.2.3) prohibits voice-based biometric comparison and allows other biometrics only together with a physical authenticator. Any process that lets a voiceprint unlock an account on its own is out of line with the current federal guideline.
• Knowledge-based questions: date of birth, address, last transaction and mother's maiden name are in breach data and on social media, and an AI caller can look them up mid-call.
• Caller ID: STIR/SHAKEN, which the FCC describes as originating carriers signing caller ID and downstream carriers validating it, helps expose spoofed calls on networks that support it. It authenticates the originating number, not the person speaking, and gives no assurance about the content of a call.
• Phone-number based one-time codes: NIST treats PSTN out-of-band authentication as restricted, tells verifiers to check risk indicators such as SIM change and number porting, and treats changing the registered number as binding a new authenticator. See the SIM Swap Fraud Guide.
• Human judgement: both FBI advisories list tells such as unusual tone or word choice, but they are guidance for individuals, not a control. Process controls must work even when the voice is indistinguishable.

Fraud controls against synthetic voice

For employees and payment processes

• Callback on a known number: both FBI advisories advise hanging up and calling back on a number found independently. Make it mandatory for any request to pay, change bank details, reset credentials or share data that arrives by phone or voice note.
• Dual control: payment-detail changes and urgent transfers need a second approver who did not take the call. See the Executive Deepfake and CEO Fraud Guide.
• Channel-switch red flag: FinCEN's deepfake alert (FIN-2024-Alert004) lists a customer trying to switch communication method during a verification check, citing technical glitches, and declining MFA as red flags. Treat a caller who pushes you onto another app or channel the same way; the May 2025 FBI advisory describes exactly that move.
• Shared verification phrases: the FBI recommends a secret word or phrase among family members. For executives' assistants and treasury teams, a pre-agreed challenge is a cheap backstop, but it is not a substitute for callback.
• Limit voice exposure: the December 2024 advisory suggests limiting public voice recordings. For executives this is rarely realistic, which is why process controls come first.

For customer-facing contact centres

• Move step-up out of the call: push an approval to the customer's authenticated app or use a passkey; do not rely on what the caller can say. See the Account Recovery and Help Desk Security Guide.
• Risk-score the call and the account together: FinCEN's red flags include access from IP addresses inconsistent with the customer profile, rapid transactions on new accounts, and immediate withdrawal of funds in ways that are hard to reverse.
• Report correctly: US financial institutions filing suspicious activity reports on suspected deepfake fraud should use FinCEN's key term FIN-2024-DEEPFAKEFRAUD, per the alert.

Securing the voice AI agents you deploy

Architecture choices and their security trade-offs

OpenAI's voice agent documentation, for example, describes the chained approach as giving the most control, including stored transcripts and policy checks before the agent responds, and the speech-to-speech approach as offering the least visibility into intermediate text. The same trade-off applies across platforms.

Identity and authorisation

• Authenticate the caller outside the model: the language model must never decide whether a caller is authenticated. A deterministic service runs the check (app push, passkey, verified session hand-off) and passes an authenticated principal to the agent. Until it does, the agent has only unauthenticated tools.
• Bind data access to the authenticated caller: tool calls should carry the caller's identity and be authorised against it, so "look up my husband's account" fails at the API, not at the prompt. See the AI Agent Authorisation Guide.
• Give the agent its own non-human identity: separate service credentials per agent and environment, short-lived where the platform supports it, held in a vault, never embedded in browser or mobile clients. See the Agentic AI Identity Guide.
• Tier the tools: read-only information tools for unauthenticated callers; account tools after authentication; payments, payee changes, credential resets and address changes only with step-up and, where the risk warrants it, a human approval. OWASP LLM06:2025 Excessive Agency describes the underlying risk of excessive functionality, permissions and autonomy.
• Plan the human handoff: transfer to a person on repeated authentication failure, on requests outside scope, and on any request the caller frames as urgent or secret. See the Human-in-the-Loop for AI Agents Guide.

Guardrails for speech input and output

• Prompt injection: OWASP LLM01:2025 Prompt Injection covers direct and indirect injection and notes that multimodal systems introduce their own injection risks. For voice, a caller can speak instructions, and retrieved content (a knowledge base article, a customer note) can carry them.
• Sensitive information disclosure: apply output filters for personal data and account details (OWASP LLM02:2025) and keep secrets out of the system prompt entirely, since system prompts can leak (OWASP LLM07:2025).
• Disclose that the caller is talking to AI: the European Commission states that the AI Act's transparency rules, which require people to be made aware they are interacting with a machine and require labelling of deepfakes, come into effect in August 2026.
• Control synthetic voices you create: if you build a custom brand voice, follow the provider's consent and disclosure rules. Microsoft, for instance, makes custom neural voice a Limited Access service, requires explicit written permission from the voice talent, requires disclosure of synthetic voices to users, and watermarks personal voice output.

Outbound AI calls and regulation

If your organisation uses AI voices to call customers, the legal position in the US changed in 2024. The FCC's Declaratory Ruling FCC 24-17 (adopted 2 February 2024, released 8 February 2024, CG Docket No. 23-362) held that the Telephone Consumer Protection Act's restrictions on calls using an "artificial or prerecorded voice" cover current AI technologies that generate human voices, including voice cloning. Practical consequences:

• Consent: such calls need the called party's prior express consent unless an emergency purpose or exemption applies, and prior express written consent where the call includes advertising or telemarketing.
• Identification: the message must identify the entity responsible for the call, and telemarketing messages must offer an opt-out.
• Enforcement: the ruling gives state attorneys general a clear basis to act against unlawful AI voice robocalls.

Outside the US, check local telemarketing and AI transparency rules before running outbound voice agents.

Voice AI observability and evaluation

Treat a voice agent as a production system that handles identity decisions. Record, for each call: call identifiers and originating number (with any STIR/SHAKEN verification result your carrier passes), authentication method and outcome, the transcript, every tool call with its arguments and the principal it ran as, guardrail decisions, handoffs and the final disposition. Store audio only where your privacy notice and retention policy allow, and protect transcripts as personal data. See the AI Agent Observability and Incident Response Guide.

Evaluation before and after launch

• Adversarial scripts: test pressure tactics (urgency, authority, distress), requests for another person's account, attempts to skip verification, and spoken instructions that try to change the agent's role.
• Synthetic voice tests: confirm that no path accepts a voice match or knowledge answers as sufficient for account changes.
• Tool-boundary tests: confirm that unauthenticated sessions cannot reach account tools at the API layer, even if the model tries.
• Regression after every change: rerun the suite after prompt, model, tool or speech component changes. Vendors' evaluation guidance, such as OpenAI's, recommends saving the audio, events, tool results and application state of test runs so failures can be replayed.

See the Red Teaming AI Agents for Identity Abuse guide for test design.

Rolling out controls

1. Map voice-initiated actions. List every process where a phone call or voice note can trigger a payment, credential reset, data disclosure or account change, human or AI handled.
2. Remove voice and knowledge as sufficient authenticators for those actions, replacing them with app approval, passkeys or callback. Decommission voiceprint-only authentication.
3. Publish staff verification rules (callback, dual approval, no codes read aloud) and train finance, help desk and executive support teams with realistic synthetic voice examples.
4. For voice agents, start read-only. Launch with information-only tools, shadow the agent against human outcomes, and add account tools only once the authentication gate and logging are proven.
5. Pilot in one queue or market, run the adversarial suite, and review transcripts daily during the pilot.
6. Keep a kill switch: each tool and flow can be disabled independently, and calls fall back to human agents. See the AI Agent Kill Switch and Containment Guide.
7. Review quarterly against new advisories, regulatory changes and evaluation results.

Decision guidance

• Small organisations: focus on process: callback on known numbers and dual approval for payment changes stop most voice fraud. Use a managed voice agent platform with information-only scope before considering account actions.
• Large organisations and regulated firms: run the chained or hybrid architecture where you need pre-response policy checks and full transcripts for audit; centralise authentication outside the agent; include voice agents in model risk and fraud governance.
• Chained versus realtime: choose realtime when conversational quality is the priority and the agent's tools are low risk; choose chained or hybrid when the agent can change accounts or move money.

Framework mapping

Common failure patterns

• Trusting a familiar voice on a call that asks for money. The Arup deepfake fraud shows how a convincing synthetic senior executive can trigger large transfers when no out-of-band check is required.
• Help desks and support lines that reset access on a phone call. Phone-based pretexting led to CRM data theft in the ShinyHunters Salesforce campaign. A voice agent with the same reset authority inherits the same weakness.
• Letting the AI decide who the caller is. An AI support assistant that sent account recovery links to attackers is the subject of the Meta AI Instagram account takeover page; authentication and recovery decisions belong in deterministic services.
• Keeping voice biometrics as a sole factor because it reduces call handling time, despite NIST's prohibition and cheap cloning.
• Deploying realtime agents with broad tools and no transcript-level logging, leaving no evidence when a call goes wrong.
• Launching outbound AI calls without TCPA consent and identification, turning a customer engagement project into regulatory exposure.

Practitioner checklist

• Inventory every voice-initiated process that can move money, reset credentials, disclose data or change accounts.
• Remove voiceprints, caller ID and knowledge questions as sufficient authentication; use app approval, passkeys or callback.
• Mandate callback on a known number and dual approval for payment-detail changes and urgent transfers.
• Run caller authentication for voice agents in a deterministic service and pass an authenticated principal to the agent.
• Give each voice agent its own vaulted, scoped credentials and tier its tools by authentication level.
• Apply input and output guardrails for injection, disclosure and system prompt leakage, mapped to OWASP LLM01, LLM02, LLM06 and LLM07.
• Disclose AI to callers and confirm consent and identification rules before any outbound AI voice campaign.
• Log transcripts, tool calls, authentication outcomes and handoffs, with retention aligned to privacy policy.
• Run an adversarial and synthetic voice evaluation suite before launch and after every change.
• Keep a tested kill switch per flow and a human fallback queue.

Standards and references

• FCC: Declaratory Ruling FCC 24-17 on AI-generated voices under the TCPA
• NIST: SP 800-63B-4 Authentication and Authenticator Management
• FBI IC3: Criminals Use Generative Artificial Intelligence to Facilitate Financial Fraud
• FBI IC3: Senior US Officials Impersonated in Malicious Messaging Campaign
• FinCEN: Alert on Fraud Schemes Involving Deepfake Media Targeting Financial Institutions
• OWASP: Top 10 for LLM Applications 2025
• European Commission: AI Act regulatory framework
• FCC: Combating Spoofed Robocalls with Caller ID Authentication

Related NHI Mgmt Group resources: Deepfake and AI Impersonation Guide · Vishing and Smishing Guide · Identity Proofing and KYC Guide · Fraud Prevention Platform Buyer's Guide · AI Agent Observability and Incident Response Guide. This guide summarises FCC, FinCEN, NIST and EU positions and is not legal advice.