AI in cybersecurity is reshaping both attack and defence

At a glance

What this is: This is an independent analysis of how AI is changing both cyberattacks and cyber defence, with the clearest finding being that the same capabilities accelerate phishing, deepfakes, recon, and credential abuse while also improving detection and response.

Why it matters: It matters to IAM practitioners because AI is now affecting authentication trust, verification workflows, and security decisions across human identity, NHI-adjacent pipelines, and SOC operations.

By the numbers:

• AI-generated phishing jumped from 4% of all reported phishing attempts in November 2025 to 56% in December 2025, a 14x increase in a single month, and has held around 40% of all reported phishing attempts through mid-2026.
• Fully AI-automated spear phishing emails achieved a 54% click-through rate, matching skilled human experts and far outperforming the 12% click-through rate of the control group.
• Organizations receive an average of 2,992 security alerts a day, with 63% going uninvestigated.

👉 Read Xygeni's analysis of AI security attacks, defence, and code risk

Context

AI in cybersecurity is the use of machine learning, natural language processing, and large language models on both sides of security operations. In this article, the primary governance gap is not model quality alone, but the speed at which AI changes trust, verification, and response workflows before teams have adapted their controls.

The same techniques that help defenders spot anomalies also help attackers write better phishing, produce deepfakes, and automate reconnaissance. That creates pressure on identity verification, approval chains, and security operations to treat AI as a control-plane issue, not just a tooling issue.

For IAM and identity leaders, the key intersection is trust: AI can create convincing human impersonation, accelerate credential attacks, and generate machine-speed decision support that still needs governance. That makes verification, escalation paths, and accountability more important than model enthusiasm.

Key questions

Q: How should security teams handle AI-generated impersonation in fraud workflows?

A: Security teams should treat AI-generated impersonation as a trust and verification problem across onboarding, recovery, and support. Stronger identity proofing, step-up verification for risky requests, and provenance checks on voice or document evidence reduce the chance that synthetic artefacts are accepted as real.

Q: Why do AI attacks change the way organisations should think about verification?

A: AI lowers the cost of believable deception and makes appearance less reliable as a signal of identity. That means organisations should rely less on what a request looks or sounds like and more on controlled verification steps, scoped approvals, and device or workflow-based confirmation.

Q: How can teams tell whether AI threat detection is improving SOC performance?

A: Look at mean time to verdict, analyst rework, and the percentage of alerts resolved with documented reasoning. If alert volume drops but analysts still have to reconstruct context manually, the platform has not changed the operating model enough to matter.

Q: What should organisations do when AI moves into code generation and software delivery?

A: Extend governance to the pipeline itself. Review AI instructions, package provenance, dependency selection, and runtime permissions before code or automation reaches production. AI-assisted development needs provenance and approval controls, not only static code scanning after the fact.

Technical breakdown

How AI changes phishing, deepfakes, and credential attacks

AI reduces the cost of personalised social engineering by generating fluent messages, cloned voices, and convincing video in volume. It also improves attacker experimentation, because models can rapidly vary wording, timing, and pretext until a target responds. Credential attacks become faster when leaked passwords, passwords patterns, and reuse habits are analysed at scale. The technical shift is from crude automation to adaptive persuasion, where the attack optimises for trust rather than malware delivery.

Practical implication: strengthen verification controls for money movement, credential resets, and executive requests before the request reaches an inbox or call.

Why AI improves detection and triage when tuned properly

Defensive AI is most effective when it correlates large volumes of logs, authentication events, and endpoint signals into a smaller set of higher-confidence incidents. This matters because many attacks do not match known signatures and instead show weak behavioural anomalies across multiple systems. In practice, AI is not replacing the analyst. It is reducing queue size, grouping related alerts, and flagging deviations from baseline fast enough for a human to decide what happens next.

Practical implication: apply AI first to alert reduction, incident correlation, and behavioural anomaly scoring rather than adding another standalone detector.

AI in software pipelines and the rise of AI supply-chain risk

AI also changes the software development attack surface. Coding assistants, agent instructions, and package selection logic can introduce unreviewed dependencies, hallucinated libraries, or malicious install paths into build pipelines. That shifts risk from pure perimeter defence to supply-chain governance inside the SDLC. The security control problem is not just scanning code after it exists, but governing the instructions and provenance that helped create it.

Practical implication: extend review, provenance, and allow-list controls to AI-assisted development workflows, not just to the final code artifact.

Threat narrative

Attacker objective: The attacker aims to convert human trust into account access, financial transfer, or operational leverage before defenders can verify the request.

1. Entry begins with AI-generated phishing, deepfake impersonation, or accelerated reconnaissance that raises the chance of a successful trust break.
2. Escalation follows when the attacker uses social proof or credential access to move from conversation to account, payment, or administrative control.
3. Impact lands in fraud, credential compromise, or faster attack execution that outpaces human review and traditional signature-based detection.

NHI Mgmt Group analysis

AI security is now an identity trust problem, not only a model-risk problem. The article’s strongest point is that AI attacks succeed by manipulating human and process trust, especially in verification-heavy moments such as payment approval and credential reset. That makes IAM and fraud controls part of the AI security conversation, not an adjacent concern. Practitioners should treat deepfake-resistant verification as core governance, not exception handling.

Detection advantage will increasingly belong to teams that can compress alert volume into decision quality. The article shows that AI’s defensive value is less about novelty and more about scale, correlation, and prioritisation. In environments where analysts already face thousands of alerts, the main value of AI is deciding what deserves human attention. Practitioners should measure whether AI reduces noise before they measure whether it adds coverage.

AI supply-chain risk is the next governance layer security teams must absorb. Once AI moves into code generation, dependency selection, and agent-assisted workflows, the threat surface shifts from the SOC into the pipeline. That is where identity governance reappears through approvals, provenance, and scoped execution rights. Practitioners should assume AI-assisted development needs the same scrutiny as any other privileged production change.

Named concept: verification trust gap. This article exposes the growing gap between what a person, approval chain, or interface appears to confirm and what is actually authenticated. The deeper the synthetic realism of AI, the less reliable visual or auditory trust becomes without out-of-band checks. Practitioners should build controls that verify intent independently of appearance.

The real differentiation is no longer who has AI, but who operationalises it with accountability. The article is clear that both attackers and defenders can access similar techniques. What separates outcomes is workflow integration, human oversight, and clear escalation. Practitioners should focus on governance models that make AI actionable without making it authoritative.

What this signals

AI is pushing security programmes toward faster verification and narrower trust windows. For identity teams, that means account recovery, payment approval, and privileged change processes need controls that assume synthetic impersonation will continue to improve.

The more AI enters development workflows, the more identity governance has to follow it into the pipeline. That makes approval boundaries, provenance, and scoped execution rights part of the same control conversation as access reviews and privilege management.

For practitioners, the next capability leap is not just AI detection, but AI governance. Teams that tie verification, escalation, and accountability together will be better positioned than teams treating AI as a standalone tool category.

For practitioners

• Implement out-of-band verification for high-risk requests Require a second channel for payment approvals, credential resets, and executive instructions that originate by voice or video. The control should verify intent separately from the medium that delivered the request.
• Use AI to reduce alert noise before expanding detection scope Measure whether AI is collapsing related alerts into fewer incidents, reducing uninvestigated queues, and improving analyst prioritisation. If it only adds another dashboard, it is not solving the operational problem.
• Extend governance into AI-assisted development workflows Review model prompts, agent instructions, package provenance, and install-time behaviour with the same discipline applied to code changes. Add approval boundaries before AI can introduce dependencies or run privileged tasks.
• Strengthen identity proofing for account recovery and reset flows Treat reset and recovery as high-risk identity events, especially where impersonation could bypass normal human intuition. Add step-up verification, callback checks, and tighter privilege on who can approve recovery.

Key takeaways

• AI is weakening the reliability of human perception as a security signal, which makes verification controls more important than ever.
• The defensive value of AI comes from triage, correlation, and decision support, not from adding another isolated detector.
• As AI moves into software delivery, identity governance must extend into prompts, agents, and runtime permissions, not stop at the inbox.

Key terms

• Deepfake: Synthetic or altered media created with AI or machine learning so that a person appears to say or do something they never did. In security terms, deepfakes are trust attacks that can distort identity verification, approval workflows, and fraud detection.
• Activation Trust Gap: The activation trust gap is the difference between trusting data because it is protected and governing it because it is being reused. It appears when organisations move data from backup or archival systems into AI pipelines without reapplying access, sensitivity, and consumer controls.
• AI supply chain: The AI supply chain is the full chain of models, datasets, prompts, tools, and vendors that influence a deployed AI system. It matters because trust cannot be assigned to the application alone. Practitioners need provenance, ownership, and dependency visibility to govern risk.
• Alert Triage: Alert triage is the process of sorting security events to decide what needs investigation, escalation, or dismissal. It is not just filtering noise. Strong triage depends on context, playbooks, and analyst judgement so that important signals are not lost in volume.

What's in the full article

Xygeni's full article covers the operational detail this post intentionally leaves for the source:

• The practical mechanics of AI-generated phishing, deepfakes, and defensive anomaly detection across real attack and response paths.
• The developer-facing risks of AI in the software supply chain, including code generation, dependency selection, and unreviewed agent instructions.
• The product-specific workflow details behind CoreAI and DevAI, including how alerts are scored, correlated, and turned into remediation actions.
• The implementation detail behind starting free with GitHub, GitLab, or Google sign-in for repository and AI scan coverage.

👉 The full Xygeni article covers the attack patterns, defensive workflow, and software pipeline risks in more operational detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to the broader security programme they already run.