AI in cybersecurity is reshaping both attack and defence

At a glance

What this is: AI is making attacks faster and more convincing while giving defenders better tools for detection, triage, and response.

Why it matters: IAM, PAM, NHI, and SOC teams need to account for AI that changes how trust is established, how alerts are handled, and how credentials are abused or defended.

Context

AI in cybersecurity is best understood as a governance problem as much as a tooling problem. The same capability can generate persuasive lures, deepfakes, and adaptive malware, while also helping defenders find anomalies and cut through alert volume. For identity programmes, the pressure point is trust: credentials, approvals, and verification flows now face machine-amplified manipulation.

The article frames a dual-use environment in which offensive and defensive teams rely on similar underlying techniques, but the operational consequences are not symmetrical. Security teams that still treat AI as a side topic will miss the way it reshapes identity verification, response sequencing, and the control points around human and non-human workflows.

A useful way to read the piece is as a warning that security controls are being asked to verify intent and authenticity under more adversarial conditions than before. That makes AI governance, alert handling, and out-of-band verification part of the same control discussion, not separate workstreams.

Key questions

Q: What breaks when AI can convincingly impersonate a real person in security-sensitive workflows?

A: Legacy trust cues break first, because many approval processes still assume a human voice, face, or writing style is enough to validate identity. When AI can mimic those cues, organisations need out-of-band verification for payments, credential changes, and other high-impact requests so legitimacy is confirmed through a separate control, not through the medium of the request itself.

Q: Why do AI-driven attacks and AI-driven defences create the same governance challenge?

A: Because both depend on the same underlying data, models, and workflow integration, the risk is not the technology alone but how reliably it is governed. If the model is poorly tuned, biased, or unauthorised, it can help the attacker as easily as the defender, so accountability, tuning oversight, and escalation rules matter as much as detection logic.

Q: How can teams tell whether AI threat detection is improving SOC performance?

A: Look at mean time to verdict, analyst rework, and the percentage of alerts resolved with documented reasoning. If alert volume drops but analysts still have to reconstruct context manually, the platform has not changed the operating model enough to matter.

Q: What should organisations do when AI moves into code generation and software delivery?

A: Extend governance to the pipeline itself. Review AI instructions, package provenance, dependency selection, and runtime permissions before code or automation reaches production. AI-assisted development needs provenance and approval controls, not only static code scanning after the fact.

Technical breakdown

Why AI-generated phishing is harder to filter than legacy spam

AI changes phishing from a low-skill volume game into a high-conviction persuasion problem. Large language models can produce context-aware language, mimic internal tone, and adapt content to specific targets, removing the obvious cues that older filters and user training relied on. That matters because detection based only on linguistic mistakes no longer scales against messages that look and sound normal. Practical implication: treat email trust as a verification problem, not a wording problem, and harden approvals around payment, credential change, and supplier instructions.

Practical implication: Shift verification controls toward out-of-band confirmation for high-risk requests, because message quality alone no longer indicates legitimacy.

How AI improves anomaly detection and alert triage

Defensive AI works by learning baseline behaviour across logs, traffic, identities, and endpoints, then flagging deviations that merit attention. Its value is not just detection but correlation, because a single malicious action often appears as many weak signals across tools. That is why AI can reduce queue noise, rank incidents, and help analysts focus on what matters. Practical implication: place AI where alert overload is already measurable, and require every model to improve analyst decision-making rather than simply add another detection source.

Practical implication: Apply AI first to triage and correlation so analysts spend less time closing false positives and more time on confirmed risk.

Why AI security now includes software supply chain and agent behaviour

The article extends AI risk beyond the SOC into development and software delivery. AI-assisted coding, hallucinated dependencies, and agent-driven actions create new trust points inside the build and deployment chain, where a model or agent can introduce unsafe artefacts before runtime controls ever see them. This is an identity issue as well as an application issue, because the agent or pipeline behaves like a non-human actor making security-relevant decisions. Practical implication: monitor AI tooling, agent instructions, and dependency intake as governed inputs, not as informal productivity aids.

Practical implication: Bring AI agents and build inputs under governed review, because unvetted machine actions can create exposure before traditional security controls engage.

Threat narrative

Attacker objective: The attacker aims to convert human trust into authorised action, stolen credentials, or financial transfer before suspicion or verification interrupts the request.

1. Entry occurs through AI-generated phishing, deepfake impersonation, or adaptive social engineering that persuades a target to trust the attacker’s request.
2. Credential or trust abuse follows when the victim authorises payment, shares access, or accepts a fraudulent identity as real.
3. Impact lands in stolen funds, credential compromise, or faster campaign scale, while defenders who rely on legacy cues lose time before containment.

Breaches seen in the wild

• Hugging Face Spaces breach 2024: Unauthorised access to Hugging Face Spaces may have exposed secrets users stored for AI apps; tokens were revoked and org tokens removed.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI security is now an identity governance problem, not only a detection problem. The article shows that the same technology can impersonate people, automate persuasion, and reshape trust decisions across security workflows. That means identity proofing, approval chains, and exception handling now sit alongside telemetry and threat hunting as core controls. Practitioners should treat authenticity as a governed control plane, not a soft human assumption.

Deepfake resilience is a process design issue before it is a model-detection issue. If an attacker can insert a convincing executive voice or face into a live request, the failure is not just technical, it is procedural. The article’s example shows that out-of-band verification must be mandatory for high-impact actions such as payments, credential changes, and supplier instructions. Teams should design for verification latency, not conversational convenience.

Alert reduction only helps when AI is tied to decision quality. The defensive promise in the article is not more automation for its own sake, but better prioritisation of what deserves human attention. That makes governance of model tuning, false positives, and escalation paths central to SOC effectiveness. Practitioners should measure whether AI is improving decisions, not whether it is merely producing more output.

Software delivery is becoming part of the AI attack surface. The article’s move from SOC use cases to AI coding agents and supply chain behaviour reflects where risk is heading. A non-human actor that writes code, suggests dependencies, or executes install actions can create exposure at build time, not just runtime. Practitioners should extend identity and policy controls to the tools that now act on behalf of developers.

What this signals

AI-driven trust abuse is collapsing the distance between social engineering and identity governance. The article shows that an attacker no longer needs malware or an exploit if they can manufacture a convincing executive identity and push a high-risk action through a live workflow. That means verification design, approval segregation, and exception handling now belong in the same governance conversation as email security and fraud controls.

Machine-speed persuasion will force teams to move verification earlier in the workflow. When a request can be generated, personalised, and iterated by a model in seconds, human judgment becomes too late if it is only applied at the final approval step. The safer pattern is to verify before the action can be executed, especially where money movement, secrets, or privileged access are involved.

For practitioners

• Embed out-of-band verification for high-risk requests Require a second channel for any payment, credential change, or privileged approval that could be manipulated by deepfake audio or video. Make the verification step mandatory and documented, not discretionary.
• Position AI in the SOC where false positives are highest Use AI to correlate alerts, suppress duplicates, and rank incidents in the queues that currently consume the most analyst time. Measure whether the model improves time-to-decision, not just detection volume.
• Govern AI agents and coding assistants as non-human actors Review agent instructions, dependency intake, and install-time behaviour as security-relevant events. Require approval paths for actions that can change code, add packages, or introduce new trust relationships.
• Expand software supply chain checks to AI-assisted workflows Inspect hallucinated dependencies, generated code, and automated package installation with the same scrutiny used for third-party libraries. Tie these checks to pipeline policy so risky artefacts are blocked before merge or release.

Key takeaways

• AI is not just changing the scale of attacks, it is changing the kinds of trust signals defenders can rely on.
• The article’s strongest evidence is that deepfake-style identity abuse can bypass human judgment without any traditional malware chain.
• Teams should respond by hardening verification, tuning AI where it reduces alert overload, and governing AI agents as part of the attack surface.

Key terms

• AI-powered social engineering: AI-powered social engineering is the use of generated text, voice, video, or interface content to manipulate a target into taking an unsafe action. The goal is not just deception, but trust transfer, where the attacker convinces a legitimate identity holder to approve, disclose, or execute something harmful.
• Deepfake-Enabled Fraud: Deepfake-enabled fraud uses synthetic media, voice, or face manipulation to impersonate real people and bypass weak verification steps. It raises the bar for identity and fraud teams because traditional checks may not reliably detect fabricated presence, especially in high-volume digital onboarding and support channels.
• Alert Triage: Alert triage is the process of sorting security events to decide what needs investigation, escalation, or dismissal. It is not just filtering noise. Strong triage depends on context, playbooks, and analyst judgement so that important signals are not lost in volume.
• Governed Non-Human Actor: A governed non-human actor is a software system that can make decisions, call tools, or affect workflows under policy constraints. The term is useful when an AI system begins to resemble a machine identity that needs access boundaries, auditability, and lifecycle oversight.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to the wider security decisions AI now influences.