Generative AI is reshaping email attacks against public agencies

At a glance

What this is: This on-demand session examines how generative AI is amplifying email-based phishing, BEC, and social engineering against state and local agencies.

Why it matters: It matters because public-sector IAM and security teams need to understand how human identity compromise begins in email, where behavioural detection can complement existing access and awareness controls.

👉 Watch Abnormal AI's on-demand webinar on AI-driven email attacks and behavioural defence

Context

Generative AI is lowering the cost and raising the quality of email-based deception. In state and local agencies, that matters because phishing and business email compromise still exploit human identity, not just mail gateways, and the payload is often trust abuse rather than malware.

The governance problem is that traditional email security and awareness controls were built for more detectable patterns of fraud. When attackers can generate convincing language at scale, defenders need stronger behavioural signals across users, inboxes, and identity-linked actions instead of relying on message inspection alone.

Key questions

Q: How should organisations reduce business email compromise risk when attackers use generative AI?

A: Organisations should stop relying on message quality as the main trust signal. Use out-of-band verification for payments, credential changes, and vendor updates, then correlate mailbox behaviour with identity and workflow telemetry so suspicious requests can be blocked even when the email looks authentic.

Q: Why do AI-generated phishing campaigns increase risk for public-sector agencies?

A: They reduce the visual and linguistic clues that users and filters once depended on, while fitting the language of agency work more convincingly. That makes inbox-based approval chains, procurement processes, and credential resets easier to abuse before anyone validates the request elsewhere.

Q: What breaks when security teams depend only on email content inspection?

A: Content-only inspection misses the behavioural evidence that usually reveals abuse, such as unusual reply timing, abnormal sender relationships, mailbox rule changes, or a suspicious payment request entering a trusted workflow. Once attackers can write better emails, the safer signal is what happens after delivery.

Q: Who should be accountable for stopping AI-driven email fraud in agencies?

A: Email security, IAM, fraud, and business process owners all share accountability because the attack crosses technical and organisational boundaries. The decisive control is whether sensitive actions require identity verification outside the inbox, not whether one team owns the entire problem.

Background and context

How generative AI changes phishing and BEC delivery

Generative AI improves volume, variation, and personalization in email attacks. Attackers can quickly produce messages that mimic agency language, procurement tone, or executive style, which reduces the telltale errors that once made phishing easier to spot. That shift does not make the attacker more autonomous, but it does make the content pipeline faster and more adaptable. The result is higher campaign throughput, more credible pretexting, and a greater chance that a single message reaches the right employee with enough context to trigger action. Practical implication: security teams should assume email lures will increasingly resemble legitimate internal correspondence.

Practical implication: update phishing simulations and detection logic for AI-generated language patterns, not just classic spoofing cues.

Why behavioural AI matters for socially engineered attacks

Behavioural AI looks for anomalies in sender behaviour, conversation patterns, account activity, and downstream actions rather than only inspecting message content. That matters because AI-assisted phishing can evade content-based rules while still producing unusual timing, relationship graphs, login paths, or payment requests. In a public-sector environment, the real signal often appears after the email arrives, when the recipient account, mailbox, or identity-linked workflow behaves differently from baseline. Practical implication: controls should correlate email activity with identity and access telemetry so responders can detect abuse even when the message itself looks legitimate.

Practical implication: correlate mail, identity, and endpoint telemetry so investigation starts with behaviour, not the text of the message.

How AI raises the operating tempo of social engineering

The core change is not only better wording, but faster iteration. AI lets attackers test subject lines, rewrite pretexts, and tune follow-up messages until one variant succeeds, which compresses the defender's response window. That increases the pressure on layered controls such as impersonation detection, mailbox protection, reporting workflows, and out-of-band verification for sensitive requests. For agencies handling payroll, grants, procurement, or vendor payments, the business risk is that a convincing thread can move from email into authorised action before a human reviews it carefully. Practical implication: reduce trust in email-originated requests for payments or credential changes.

Practical implication: require out-of-band verification for payment, account, and credential-change requests originating in email.

NHI Mgmt Group analysis

AI-assisted email fraud is a human identity problem first, not an email filter problem. The article centres on phishing, BEC, and social engineering, all of which succeed by manipulating human trust and workflow dependence. That means the control gap is not only message detection but also how organisations authenticate intent when a request arrives by email. Practitioners should treat mailbox compromise and impersonation as identity events, not just messaging events.

Behavioural detection is becoming the more durable control plane for social engineering defence. Content inspection alone struggles when attackers can generate fluent, context-aware messages at scale. A behavioural layer can watch for unusual sender relationships, conversation timing, reply-chain anomalies, and suspicious follow-on actions across identity-linked systems. Practitioners should align email telemetry with identity and access observability.

Public-sector environments are especially exposed because email still drives high-trust workflows. State and local agencies often route procurement, benefits, and operational approvals through inbox-based processes that are easy to impersonate. When AI lowers the cost of tailored pretexts, the attack surface expands beyond users to the business processes they execute. Practitioners should reassess which email requests can still be trusted at face value.

AI-generated social engineering increases the value of identity verification outside the inbox. The article's message is that defenders need to stop treating the email itself as proof of legitimacy. Stronger call-backs, secondary approval paths, and workflow-level validation are now central to reducing fraud risk. Practitioners should design verification around the action, not the message.

Content-aware and behaviour-aware defence should be paired, but behaviour must be the decisive signal. Generative AI will continue to improve message quality, which means defenders cannot win by pattern-matching prose alone. The better programme question is whether suspicious email is changing human behaviour, account behaviour, or transaction behaviour in ways the baseline should never permit. Practitioners should measure that end-to-end chain.

From our research:

• 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate, according to AI Agents: The New Attack Surface report.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation, according to SailPoint.
• For broader context on non-human identity governance, see NHI Lifecycle Management Guide and the 52 NHI Breaches Analysis.

What this signals

Email fraud is increasingly an identity governance issue, not just a security awareness issue. As AI improves the realism and scale of social engineering, agencies need controls that validate intent across the workflow, not just the mailbox. The governance signal is clear: the inbox can no longer be the final trust boundary for payments, account changes, or vendor updates.

Behavioral evidence will matter more than message inspection as attacks become more fluent. The safest programmes will connect mail telemetry to identity events, privilege changes, and downstream business actions. That creates a practical route for prioritising suspicious requests even when the content itself is polished and context-aware.

Fraud-resistant workflow design now belongs in IAM planning. Teams that already struggle with approval sprawl, weak separation of duties, or over-trusted email-based processes will feel the pressure first. Practitioners should treat AI-generated phishing as a trigger to harden verification paths across human identity and access governance.

For practitioners

• Tighten verification for email-originated requests Require out-of-band confirmation for payment instructions, credential resets, bank detail changes, and procurement approvals that arrive by email. Make the approval path depend on a second channel tied to the requester’s known identity, not reply-chain legitimacy.
• Correlate mail and identity telemetry Feed mailbox activity, login behaviour, and downstream workflow events into the same detection pipeline so suspicious email can be evaluated by what the recipient account does next. Look for abnormal forwarding, impossible travel, and unusual approval timing.
• Tune detections for AI-written pretexts Refresh detection logic and phishing simulations to account for fluent, context-rich language with fewer grammar errors and more convincing executive tone. Focus on relational anomalies and transaction risk rather than simple keyword or style flags.
• Restrict high-risk email workflows Place extra controls on mailbox rules, vendor changes, payroll updates, and grant or invoice actions that are frequently abused in BEC campaigns. Limit who can trigger these workflows and require additional review when the request originates from email.

Key takeaways

• Generative AI is making phishing and BEC more convincing, which shifts the defence problem from spotting bad writing to validating trust and intent.
• Public-sector agencies are especially exposed because email still powers high-trust workflows that attackers can exploit at scale.
• Behavioural detection, out-of-band verification, and tighter workflow controls are now essential for reducing AI-assisted social engineering risk.

Key terms

• Business Email Compromise: Business Email Compromise is a form of fraud where attackers use deceptive email to induce payments, credential changes, or data transfers. It succeeds by impersonating trusted relationships and exploiting approval workflows, making identity verification and transaction controls more important than message authenticity alone.
• Behavioural Detection: Behavioural detection identifies suspicious activity by comparing actions, timing, and relationships against normal patterns. In email security, that means looking beyond message content to mailbox behaviour, login signals, forwarding rules, and downstream workflow actions that reveal abuse after delivery.
• Social Engineering: Social engineering is the manipulation of people into taking actions they should not take, such as sharing credentials or approving payments. In identity programmes, it is a governance problem as much as a security problem because it targets trust, process, and accountability.
• Out-of-Band Verification: Out-of-band verification confirms a request through a separate trusted channel rather than the channel used to deliver the request. For high-risk identity and financial actions, it reduces reliance on email as proof of legitimacy and helps stop impersonation-based fraud.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.

This post draws on content published by Abnormal AI: generative AI and email attack tactics in state and local agencies. Read the original.