Inference time manipulation is an attack that changes a model’s behaviour while it is generating outputs, without altering its training data or internal parameters. For API accessed LLMs, this is especially important because the attacker may only need crafted prompts to influence the result.
Expanded Definition
Inference time manipulation is the class of attack where an adversary changes a model’s behaviour while it is generating output, rather than changing training data or model weights. In practice, that means the system can be steered at runtime through crafted prompts, adversarial instructions, tool outputs, retrieved context, or other inputs that shape the next token decision. The core boundary is important: the model may still be the same model, but the inference session is being influenced.
This term sits close to prompt injection, context poisoning, and tool misuse, but it is broader than any one of them. Prompt injection is one way to manipulate inference time; it is not the whole category. The same idea can apply to chatbots, retrieval-augmented systems, agentic workflows, or any API-exposed model where output depends on mutable runtime context. Security teams should treat the inference path as part of the trust boundary, not as a passive transport layer.
A useful authority reference for adversarial AI technique taxonomy is MITRE ATLAS adversarial AI threat matrix, which helps separate runtime manipulation from training-time compromise.
Examples and Use Cases
- A customer support chatbot is given a hidden instruction in retrieved content that overrides the intended response policy.
- An agentic workflow reads a malicious tool result and follows attacker-controlled text as if it were trusted operational guidance.
- A retrieval-augmented assistant is fed poisoned context that biases the answer toward a false summary or unsafe action.
- An API-accessed model is prompted to ignore guardrails and disclose data that should have stayed outside the response scope.
- A multi-step assistant is manipulated through conversation state so that the model carries forward an attacker’s framing into later turns.
These cases differ in mechanism, but they share the same operational reality: the attacker is not necessarily changing the model itself, only what the model sees at inference time. That is why a system can look intact while still producing unsafe or unreliable output.
In more complex agent workflows, the tradeoff is that richer context improves utility, but every additional input source also expands the surface an attacker can influence.
Security Implications
When inference time manipulation is misunderstood, defenders often overfocus on model training and underprotect the runtime path. The result is that a model can be “securely built” yet still be pushed into unsafe, misleading, or policy-violating behaviour during live use. The practical consequence is not only bad answers, but bad decisions when output drives automation, approval, classification, or user-facing actions.
The blast radius depends on what the model is allowed to do after it is influenced. If output is merely informational, the damage may be limited to deception or data exposure. If output triggers tool calls, ticket creation, code generation, or access decisions, a manipulated inference session can become an execution path. That makes monitoring and containment of prompts, retrieved content, tool outputs, and conversation state part of the security control surface.
A common practitioner signal is unexplained output drift that tracks attacker-controlled wording, especially where the model appears to “obey” content that should have been treated as untrusted.
Security, Operational and Governance Implications
Inference time manipulation matters because it changes how organisations should design trust boundaries around AI systems. The key governance issue is that runtime inputs are not all equal, and systems that mix user text, retrieved documents, logs, and tool responses need clear rules about which sources may influence the model and which must be treated as hostile.
Operationally, the strongest controls are usually contextual: limit what the model can act on, segment untrusted content, constrain tools, and log the path that led to a sensitive output. For agentic systems, the question becomes whether the model is merely summarising content or is allowed to act on manipulated content with real side effects. That distinction changes incident response, review, and approval thresholds.
For broader AI governance, frameworks such as NIST AI Risk Management Framework and OWASP Top 10 for Agentic Applications 2026 are useful because they tie runtime manipulation to risk management, tool abuse, and control design rather than treating the model as a static artifact.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 address the attack and risk surface, while NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | GOVERN — AI Governance | Defines governance for managing AI risks, including runtime manipulation. |
| MEASURE — Measure AI Risks and Impacts | Supports measuring manipulation risk and monitoring model behaviour under attack. | |
| Recommendation — Assign governance for runtime AI risks and define accountability for model misuse. Measure manipulation scenarios and track model behaviour under adversarial inputs. | ||
| OWASP Agentic AI Top 10 | Prompt Injection | Directly covers runtime instruction hijacking and context steering in agentic systems. |
| Recommendation — Filter and isolate untrusted instructions before they can alter agent behaviour. | ||
