Why do general-purpose LLMs struggle with offensive security work?

Why This Matters for Security Teams

General-purpose LLMs can be impressive at summarising tools, generating hypotheses, and translating language across domains, but offensive security work depends on something more demanding: deliberate adversarial reasoning under uncertainty. That means revisiting assumptions, testing weak signals, and persisting after the first failed path. Guidance from the NIST AI Risk Management Framework is useful here because it treats AI behaviour as a managed risk, not an automatically reliable capability.

For security teams, the failure mode is not that an LLM cannot name techniques or explain common attack paths. The deeper issue is that it often lacks stable intent, memory of failed hypotheses, and a reliable internal model of what evidence should disprove a claim. In offensive work, those gaps matter when the task is to infer hidden privilege paths, chain low-confidence signals, or validate whether a suspected weakness is actually exploitable. The model can sound confident while still missing the operational detail that matters most.

That is why frameworks such as the MITRE ATLAS adversarial AI threat matrix and OWASP Agentic AI Top 10 are relevant even when the question is about security testing rather than AI governance. They help teams distinguish between language competence and adversarial reliability. In practice, many security teams encounter these limits only after an LLM has confidently suggested a plausible path that falls apart during validation, rather than through intentional red-team style evaluation.

How It Works in Practice

Offensive security work is iterative. A practitioner usually starts with incomplete data, forms a hypothesis, probes for validation, then revises the model of the target environment based on what fails. General-purpose LLMs are not naturally optimized for that loop. They are better at producing a single coherent answer than at managing a sequence of adversarial decisions where the next step depends on failure analysis.

There are several practical reasons for this:

• They are trained to continue text plausibly, not to maximise exploit discovery or proof quality.
• They can overweight common patterns and underweight sparse clues that matter in niche environments.
• They may not preserve the operational state needed to compare alternate paths or revisit earlier assumptions.
• They often need external tools, sandboxing, and human validation to avoid false confidence.

That is why current guidance suggests treating LLMs as assistive analysts, not autonomous operators, especially where the work involves chaining primitives, checking exploit preconditions, or deciding whether an attack path is worth pursuing. The NIST AI 600-1 Generative AI Profile is helpful because it frames these systems around trustworthy use, including output validation and human oversight. It is also sensible to align test workflows with the NIST SP 800-53 Rev 5 Security and Privacy Controls for logging, change control, and verification, especially when AI-generated advice influences operational decisions.

In operational terms, teams usually get better results when the LLM is constrained to narrow sub-tasks such as summarising findings, mapping known weaknesses to techniques, generating test checklists, or comparing evidence against a predefined rubric. The human tester still owns the exploit logic, the validation steps, and the final judgment. These controls tend to break down when the environment is highly novel, the evidence is sparse, or the workflow requires persistent adversarial reasoning across many dependent steps.

Common Variations and Edge Cases

Tighter control over AI-assisted offensive work often increases process overhead, requiring teams to balance speed against evidentiary quality. That tradeoff is worth it because the point is not to forbid LLM use, but to avoid mistaking fluent output for tested security reasoning.

There is no universal standard for how much autonomy is safe in offensive workflows yet. Best practice is evolving, especially where AI agents are allowed to call tools, browse internal data, or coordinate multi-step analysis. In those environments, the line between helpful assistant and unreliable operator becomes important, which is why the CSA MAESTRO agentic AI threat modeling framework and OWASP Top 10 for Agentic Applications 2026 are useful reference points.

Edge cases also appear when the model is used for malware analysis, exploit-chain brainstorming, or post-exploitation planning. In those cases, the model may be able to describe known patterns, but it still lacks the robust verification discipline needed to separate a neat idea from a working path. The Anthropic report on the first AI-orchestrated cyber espionage campaign shows why orchestration and human oversight remain central even when AI appears capable of complex workflows. For security organisations, the practical answer is to use LLMs to accelerate analysis, not to outsource adversarial judgment.

Related resources from NHI Mgmt Group

• When should organisations choose purpose-built security platforms over general tools?
• What should organisations check when a security vendor also uses general-purpose AI tools?
• Why do LLMs struggle with security review as codebases get larger?
• Why do general-purpose LLMs miss vulnerabilities that human testers still find?