Why do in-context learning systems create more risk than static LLM use?

Why in-context learning is more volatile than static LLM use

In-context learning changes the effective behaviour of the system at inference time. Static LLM use is comparatively bounded by the model and a known application layer, while in-context learning lets examples, instructions, and adjacent content reshape the task on the fly. That makes the session itself part of the control surface, so the security question becomes how the model is influenced during use, not just who can reach it.

The practical difference is that the system can be steered by whatever arrives in context, including malformed prompts, contradictory instructions, copied secrets, or content that was never intended to be operational input. A static deployment can still be unsafe, but in-context systems add a second layer of variability because the inference session can alter interpretation without a code or weight change.

That volatility also changes how teams think about trust. If the context is untrusted, then the model may follow the wrong instruction hierarchy, reproduce sensitive material, or carry forward assumptions from earlier turns. Security teams therefore need to treat prompt content, retrieval output, and attached files as inputs that can change control outcomes, not just as user text.

Where the extra risk comes from in practice

The risk is not merely that the model makes mistakes. The larger issue is that context can become an attack path: prompt injection, data poisoning, example poisoning, and hidden instructions can all shape the next response or action. In an in-context setting, the attacker does not need to alter the base model to influence behaviour, only the runtime context the model uses to infer the task.

That matters because context often crosses trust boundaries. A user prompt may be mixed with retrieved documents, previous chat history, tool output, or copied text from an external source. If those sources are not filtered and segmented, a malicious or malformed snippet can override the intended task, leak adjacent content, or induce unsafe tool use.

Static LLM use generally has a narrower blast radius because the behaviour is closer to a fixed product configuration. In-context learning expands the attack surface to include the session state itself, especially where long context windows, shared conversations, or retrieval-augmented workflows allow one input to influence later decisions.

What governance has to cover when context becomes an input channel

Governance has to extend beyond model access and API permissioning to the integrity of the live session. That means defining which sources may enter context, how they are labeled, when they expire, and which content types are blocked from becoming instructions. It also means deciding what the model is allowed to do with context-derived instructions, especially when downstream actions touch data, tools, or external systems.

For practitioners, the key control shift is from “who can call the model” to “what can influence the model once it is called.” A system can have strong identity controls and still be exposed if untrusted context can change the model’s interpretation of the task. That is why session-level guardrails, instruction hierarchy, content sanitization, and retrieval filtering belong in the risk model.

When the use case includes tool calls or workflow automation, the issue becomes sharper. If the model can act on context, then a poisoned prompt can become an authorization bypass by proxy, especially if the agent is allowed to infer intent from examples rather than from a tightly defined policy.

Risk and Threat Considerations

In-context learning increases exposure because the attacker only needs to influence the session, not the model weights. That creates a broader set of failure modes: prompt injection, malicious examples, contaminated retrieval, and leakage of sensitive text already present in the conversation.

Failure mechanism: Untrusted context is interpreted as task guidance, so hostile or malformed instructions can override the intended objective, steer tool use, or surface adjacent secrets.

Impact: The system can produce unsafe outputs, disclose protected information, or execute actions that look legitimate from the model’s point of view but were never approved by the operator.

Practitioner Guidance

What to prioritise: Treat the session boundary as a security boundary. If a prompt, retrieval source, or file can influence downstream decisions, classify it as a control input and review its provenance before it reaches the model.

What to verify: Confirm that examples, retrieved passages, and chat history are isolated by trust level and cannot silently rewrite system instructions. If the system cannot distinguish instruction from content, assume the session is steerable.

Common mistake: Teams secure the model endpoint and assume they have secured the use case. For in-context systems, the dominant risk often sits in the path into the context window, not in the model itself.

Practitioner takeaway: Static model access is only part of the problem; once the model can learn from context at runtime, the control objective becomes preventing untrusted input from becoming de facto policy.

Related resources from NHI Mgmt Group

• Why do static LLM guards create more risk when attackers use evolving jailbreak techniques?
• Why do AI agents create new risk in non-human identity management?
• When does AI agent access create more risk than it reduces?
• When do AI agent credentials create more risk than they reduce?