Generative AI is shifting cybersecurity from summaries to execution

At a glance

What this is: This is Torq’s analysis of how generative AI and agentic AI are changing SOC operations, with the key finding that defenders need governed execution, not just AI summaries.

Why it matters: It matters to IAM and security teams because AI-driven workflows introduce new identity, access, and accountability questions around who or what can act, approve, and audit response actions across NHI, autonomous, and human-operated processes.

By the numbers:

• AI-generated phishing emails now achieve a 54% click-through rate.
• Organizations that extensively use AI and automation saved an average of $1.9 million per breach and reduced their breach lifecycle by 80 days.
• 94% of organizations were actively planning or testing generative AI for specific security use cases.

👉 Read Torq's analysis of generative AI, agentic AI, and the AI SOC

Context

Generative AI has lowered the cost of creating convincing phishing, social engineering, and malware content, which means the defensive baseline has changed. The first-order problem is no longer whether attackers can scale abuse, but whether security operations can keep pace with AI-assisted volume and speed. In practice, the cybersecurity controls that once depended on manual triage, static playbooks, and analyst time are now under pressure from machine-generated campaigns that adapt quickly.

The article’s core governance issue is the gap between AI that summarises and AI that executes. That matters because security teams are increasingly introducing AI into workflows that touch non-human identities, privileged actions, and response decisions, which means auditability, approval boundaries, and human-on-the-loop design are now operational requirements rather than theoretical concerns.

Key questions

Q: How should security teams govern AI-assisted actions in the SOC?

A: Security teams should treat AI-assisted SOC actions as policy-governed machine behavior, not informal automation. Define which tools the system may access, which actions require approval, and what must be logged for later review. The goal is to keep investigation speed while preserving human accountability and least privilege across prompts, queries, and remediation steps.

Q: Why do AI phishing attacks create more risk than traditional phishing?

A: AI lowers the cost, time, and skill needed to produce personalised lures, so attackers can run more campaigns and iterate faster. That increases both exposure and realism. The result is a higher probability that a target will trust a message long enough to hand over credentials or payment information.

Q: What breaks when generative AI is allowed to execute security actions without governance?

A: The organisation loses traceability, consistent decision boundaries, and clear accountability for remediation steps. A model can appear helpful while still taking actions that exceed intent or policy. Without scoped permissions and review points, incident handling becomes fast but difficult to audit or reverse.

Q: What should organisations test before adopting agentic AI in security operations?

A: Organisations should test whether the agent can act safely under failure, whether its actions are traceable, and whether an incorrect decision can be rolled back. The key question is not only what the agent can do, but what happens when upstream telemetry is wrong or incomplete. Without that test, automation can spread error faster than humans can correct it.

Technical breakdown

Why generative AI changes the phishing threat model

Generative AI compresses the effort required to craft believable phishing at scale. Large language models can tailor language, tone, and context to specific recipients, which reduces the signal quality that older detection approaches relied on, such as obvious grammar errors or generic pretexts. The result is not just more phishing, but more persuasive phishing that blends into normal business communication. Defenders have to treat content generation as an attacker acceleration layer, not merely a messaging tool.

Practical implication: security teams need behaviour-based and identity-aware email controls, not just content filters.

Why SOC summarisation is not the same as autonomous execution

Generative AI can enrich alerts, summarise incidents, and recommend next steps, but it does not inherently own the workflow end to end. Agentic AI adds goal-setting, planning, decisioning, and action execution, which is why it changes the control problem. Once a system can decide which tool to use, when to trigger it, and how to progress a case, the question becomes who governs that runtime authority and how those actions are constrained, logged, and reversible.

Practical implication: organisations need explicit controls for tool permissions, action scopes, and approval thresholds before allowing AI to operate in the SOC.

How AI governance differs from classic automation governance

Traditional automation follows deterministic rules. AI-driven operations introduce probabilistic reasoning, which makes traceability, policy enforcement, and model oversight more important. In a security operations context, that means every case closure, quarantine action, escalation, and remediation step should be attributable to a documented policy path or human review point. Without that, AI can become a fast but opaque decision layer inside an otherwise governed environment.

Practical implication: security leaders should require audit trails, model governance, and rollback paths for every AI-assisted response action.

Threat narrative

Attacker objective: The attacker’s objective is to exploit human trust and machine speed together, turning AI-assisted social engineering into faster credential theft, fraud, or operational disruption.

1. Entry occurs when attackers use generative AI to produce highly personalised phishing lures or deepfake-assisted social engineering that bypasses ordinary suspicion thresholds.
2. Escalation happens when a victim credential, token, or approval path is captured and used to reach systems or workflows that were not intended to be exposed.
3. Impact follows when the attacker uses that trusted access to move faster, increase dwell efficiency, or trigger fraudulent actions at scale.

NHI Mgmt Group analysis

Generative AI has become a force multiplier for identity abuse, not just a content engine. The article correctly shows that attackers are using AI to make phishing more convincing and faster to deploy. The governance lesson is that identity verification controls are being tested at the point where human judgment meets synthetic persuasion, which creates a boundary problem for IAM, fraud, and SOC teams. Practitioners should treat AI-assisted social engineering as an identity assurance issue, not only an email security issue.

Execution is the new control surface. Summaries do not change the security outcome if the organisation cannot act on them consistently and safely. Once AI begins selecting tools, progressing investigations, or triggering remediation, the programme needs clear privilege boundaries, human approval thresholds, and auditable action trails. That shifts the centre of gravity from alert quality to governed runtime authority, which is where identity and PAM discipline become directly relevant.

AI SOC design now depends on non-human identity governance. When an AI system can call tools, query data, and initiate response steps, it behaves like a privileged service actor and should be managed accordingly. Runtime delegation debt: the longer organisations let AI systems inherit broad tool access without explicit scoping, the harder it becomes to explain or contain their actions. Practitioners should map AI action rights the same way they map high-risk service account access.

Phishing resistance is increasingly a policy problem, not just a user-awareness problem. The article’s attack examples show that better wording alone can bypass the weak spots in people-centric defences. Organisations need stronger controls around identity proofing, transaction confirmation, and anomalous request handling, especially where AI can imitate familiar business context. The practical conclusion is to harden workflows, not just train users.

The market is moving from copilots to orchestrators because security teams need outcomes, not summaries. That does not remove governance risk. It raises the bar for lifecycle controls, authorisation policy, and evidence retention across machine-driven actions. The organisations that benefit most will be the ones that can govern AI as an operational actor, not the ones that merely add it as a reporting layer.

What this signals

Runtime delegation is becoming the real governance challenge. As security teams let AI systems query data, enrich incidents, and trigger actions, the question stops being whether AI can help and becomes who can authorise its next move. That is a direct identity and privilege management problem, especially where service accounts, tokens, and tool permissions underpin the workflow.

AI governance debt will accumulate where action boundaries are unclear. If teams cannot distinguish summarisation from execution, they will eventually grant models broad operational reach by accident. The practical response is to define narrow action scopes now, then align them with the NIST AI Risk Management Framework and the MITRE ATLAS adversarial AI threat matrix when AI touches security operations.

Human verification controls will need to absorb synthetic trust abuse. Deepfake-enabled fraud and AI-authored phishing mean that many existing trust checks are too text-based or too static. Security, IAM, and fraud teams should prepare for stronger confirmation workflows, more explicit identity proofing, and tighter handling of high-risk requests that originate through digital channels.

For practitioners

• Implement governed tool access for AI workflows Restrict which systems an AI SOC workflow can query, modify, or remediate. Use least privilege for every tool integration and separate read-only enrichment from write-capable actions so that one model cannot inherit blanket authority across the security stack.
• Require human-on-the-loop approval for high-impact actions Define which response steps may run automatically and which must pause for analyst confirmation, especially quarantine, account disablement, ticket closure, and external notifications. Keep the approval boundary explicit and review it as the workflow matures.
• Harden phishing workflows with identity-aware verification Treat suspicious requests as an identity validation problem as well as a content problem. Add out-of-band confirmation for payment, access, and executive requests, and tune controls for deepfake voice, impersonation language, and anomalous sender behaviour.
• Log every AI-assisted decision and action Capture the prompt, model output, policy decision, human override, and downstream action for each incident workflow. That record is what makes AI operation auditable, supports investigations, and reduces the chance of opaque automated change.
• Separate summarisation from execution pipelines Use one path for generating context and a different, more tightly controlled path for taking action. This reduces the risk that a summarising model is implicitly trusted to execute tasks beyond its intended scope.

Key takeaways

• Generative AI is making phishing and social engineering cheaper, faster, and more convincing, which shifts the defensive problem toward identity assurance and workflow verification.
• The evidence points to real operational value when AI is governed well, but the same systems can create accountability gaps if they are allowed to act without scoped authority.
• Security teams should separate AI summarisation from AI execution, then apply least privilege, auditability, and human approval where the action carries risk.

Key terms

• Agentic AI: Autonomous AI systems capable of planning, deciding, and taking actions — including calling APIs, writing code, and orchestrating other agents — with minimal human oversight. Agentic AI introduces new NHI risks as agents must authenticate to external services.
• Human-on-the-loop: A control model where AI handles routine decisions while a human supervises exceptions and high-risk cases. In identity governance, it reduces manual effort without removing accountability, but only when escalation criteria, evidence capture, and approval boundaries are clearly defined and consistently enforced.
• Runtime Delegation: The process by which an identity is allowed to choose actions, tools, or next steps while a task is in progress. In AI agent environments, runtime delegation is risky when it is broad, opaque, or disconnected from explicit policy, because the resulting behaviour may exceed the original intent.
• Identity Assurance: The confidence an organisation has that a person or system is truly who it claims to be before access or action is granted. In modern IAM, assurance depends on evidence quality, channel trust, and the strength of verification around high-risk decisions.

What's in the full article

Torq's full article covers the operational detail this post intentionally leaves for the source:

• Step-by-step SOC workflow examples for AI-assisted triage and response
• Specific guidance on where agentic AI differs from generative AI in operational execution
• Implementation detail on human-on-the-loop controls and auditability
• Torq's own architecture framing for AI SOC orchestration and case handling

👉 Torq's full post covers the AI SOC workflow detail, governance framing, and execution model in more depth.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management in a way that helps security practitioners align identity controls with operational risk. It is a practical fit for teams managing machine access, privileged workflows, and emerging AI governance requirements.