Prompt Attack

Expanded Definition

Prompt attack is a broad, evolving term for attempts to manipulate an AI system through crafted input that changes behaviour, bypasses guardrails, or exposes restricted information. In practice, the prompt may target a chat interface, an embedded assistant, or an agentic workflow that can call tools, query data, or trigger actions. That makes the risk larger than simple prompt wording: the real issue is whether the system accepts untrusted input as if it were instruction authority.

Definitions vary across vendors and research communities, but the core security pattern is consistent. Some usage focuses on direct prompt injection, while broader use includes indirect attacks hidden in retrieved content, files, webpages, or tool outputs. For a threat-model view, MITRE ATLAS adversarial AI threat matrix is useful for mapping attack patterns, while defensive programs often align the surrounding control environment to NIST SP 800-53 Rev 5 Security and Privacy Controls.

The most common misapplication is treating prompt attack as a user-interface nuisance, which occurs when organisations ignore indirect injection paths and tool-enabled execution.

Examples and Use Cases

Implementing prompt-attack defenses rigorously often introduces friction, because stronger filtering, context partitioning, and output checks can reduce model flexibility and increase false positives, requiring organisations to weigh safety against usability and speed.

• A support chatbot is tricked into revealing internal policy text after a user includes instruction-like content that overrides the assistant’s intended system rules.
• An AI agent summarises a webpage that contains hidden instructions, then uses its own tool access to send data to an external endpoint, a pattern discussed in adversarial AI reporting such as the Anthropic — first AI-orchestrated cyber espionage campaign report.
• A retrieval-augmented generation workflow ingests poisoned documents, and the model follows embedded instructions that were never meant for the end user.
• A security copilot receives adversarial wording intended to steer it toward disclosing incident-response notes, credentials, or internal ticket data.
• A red team tests whether a model can be induced to ignore safety constraints and generate phishing content, malware-adjacent guidance, or deceptive summaries.

These scenarios also overlap with classic intrusion workflows described in the MITRE ATT&CK Enterprise Matrix, where social engineering, execution, and exfiltration can be enabled through AI-mediated interaction.

Why It Matters for Security Teams

Prompt attack matters because it converts natural-language interaction into a control surface. If the AI can access files, APIs, identity data, or downstream automations, a successful prompt attack can become an authorization problem, a data exposure problem, and an operational integrity problem at the same time. That is especially important for agentic AI, where model output may drive tool use without a human re-checking every step.

Security teams need to distinguish between input sanitisation, instruction hierarchy, access control, and action approval. No single standard governs prompt attack as a standalone term yet, so programs should treat it as a composite risk spanning secure design, content filtering, least privilege, and human oversight. Guidance from CISA cyber threat advisories helps teams connect AI abuse patterns with broader operational threat awareness, while NIST control families provide the governance backbone for logging, access restriction, and system integrity.

Organisations typically encounter the consequences only after an assistant leaks data, triggers an unsafe action, or is used in an active intrusion, at which point prompt attack becomes operationally unavoidable to address.

Related resources from NHI Mgmt Group

• Who is accountable when a prompt bombing attack succeeds?
• Why does Agentic AI make NHI attack surface expand so significantly?
• What is the 'no prompt means no action' principle in Agentic AI security?
• What is the difference between attack surface management and NHI governance?