Context Engineering Explained: Prompts, Memory, Retrieval

Context engineering is the practice of deciding which tokens a language model sees on each call: the system prompt, tool definitions, retrieved documents, memory, examples and conversation history, assembled and trimmed so the model has what it needs and little else. Anthropic's engineering team describes it as the natural progression of prompt engineering, because an agent that runs for many turns has to re-curate its context every time it calls the model, not just once. The term matters because long-running agents fail less from bad wording than from overloaded, stale or contradictory context. This page explains the parts of a context window, the main techniques (retrieval, memory, compaction, sub-agents), the context management features documented by the major model APIs, and how to measure whether your approach works. It is written for learners first, then for engineers building agents.

Last checked: 2026-11-15. Providers are listed alphabetically; order is not a ranking.

This is an independent reference page from NHI Mgmt Group based on public provider documentation; it is not sponsored or endorsed by any provider.

Key takeaways

Context is a budget, not a bucket. Chroma's Context Rot report tested 18 models and found performance becomes less reliable as input grows, even on simple tasks, so adding more tokens is not free.
Prompt engineering is one slice. The system prompt is only one input; tools, retrieved data, memory files and message history usually take far more of the window in an agent.
Four moves cover most techniques. LangChain groups them as write (save outside the window), select (pull in what is relevant), compress (summarise or trim) and isolate (split work across separate contexts).
The APIs now do some of this server-side. Anthropic documents context editing and compaction, Google documents implicit caching and long-context practice, OpenAI documents compaction and the Conversations API; each works differently.
Everything in context is an input the model may obey. Retrieved pages, tool output and memory are untrusted data, which is why context engineering and prompt injection defence have to be designed together.

What is context engineering?

A model has no memory between API calls. Whatever it "knows" about the task on a given call is whatever sits in its context window at that moment. Context engineering is the work of building that window: choosing sources, ordering them, deciding what to drop and keeping the total inside the model's limit. For what a context window is and how large current windows are, see Context Windows Explained.

Anthropic's September 2025 engineering post frames the core question as which configuration of context is most likely to produce the behaviour you want, and treats curation as iterative: it happens every time the agent decides what to pass to the model. That is the practical difference from prompt engineering.

Aspect	Prompt engineering	Context engineering
Main object	The instruction text, especially the system prompt	The whole token set on each call: instructions, tools, data, memory, history
When it happens	Mostly at design time	At design time and on every turn of a running agent
Typical failure	Ambiguous or brittle wording	Overloaded, stale, contradictory or poisoned context
Main levers	Wording, structure, examples	Retrieval, tool design, memory, compaction, sub-agents, caching layout
Who owns it	Prompt author	The agent harness and the engineers who build it

What goes into a context window?

Every agent call is assembled from some mix of the components below. Naming them separately helps you see where tokens go and which ones you control.

Component	What it is	Design guidance from provider docs
System prompt	Role, rules, output format	Anthropic advises the "right altitude": neither hardcoded if-else logic nor vague guidance; organise sections with XML tags or Markdown headers; start minimal and add instructions in response to observed failures
Tool definitions	Names, descriptions and parameter schemas for functions the model can call	Keep tools self-contained, clearly distinct and token-efficient in what they return; avoid overlapping tool sets
Examples	Few-shot demonstrations	A small set of diverse, canonical examples rather than a long list of edge-case rules (Anthropic); Google notes many-shot prompting with hundreds or thousands of examples as an option with long context
Retrieved data	Documents, search results, database rows, file contents	Load what the current step needs; references (paths, queries, links) can stand in until the data is needed
Memory	Facts or notes saved from earlier sessions or earlier in a long task	Stored outside the window and read back selectively (see the memory section below)
Conversation and tool history	Earlier user turns, model replies, tool calls and results	Usually the fastest-growing part in agents; the main target for clearing and compaction
Reasoning content	Thinking or reasoning items from earlier turns, on models that produce them	Provider-specific rules on whether to keep or drop them between turns
The user's request	The current question or task	Google advises placing the query at the end of the prompt, after the other context, especially with long inputs

Through the Model Context Protocol, much of this context can come from external servers. The MCP specification (version 2025-11-25) defines three server features: resources (context and data for the user or model), prompts (templated messages and workflows) and tools (functions for the model to execute). See the MCP Security Guide for how to govern those connections.

Why more context is not always better

Large windows (1M tokens on several current models) make it tempting to send everything. Three published findings argue against it:

Context rot: Chroma's July 2025 report tested 18 models, across Anthropic Claude, Google Gemini, OpenAI GPT and Alibaba Qwen3 models, and concluded that models do not use their context uniformly. Performance dropped faster with length when the question and the answer were less similar, and topically related distractors lowered accuracy further.
Attention budget: Anthropic describes each new token as drawing down a finite attention budget, with diminishing returns as the window fills.
Multiple facts are harder than one: Google's long-context guide notes that retrieval accuracy drops when a prompt holds several "needles" rather than one, and that longer inputs increase time to first token.

A widely reproduced taxonomy in LangChain's July 2025 post names four ways context goes wrong:

Failure	What happens	Typical fix
Context poisoning	A hallucination or false fact enters context and persists across turns	Validate before writing to memory; let memory entries expire; recompute from source
Context distraction	The context grows so large it overwhelms what the model learned in training	Compaction, clearing old tool results, sub-agents
Context confusion	Superfluous content, such as unused tool definitions, steers the response	Fewer, sharper tools; select tools per task
Context clash	Parts of the context contradict each other	Single source of truth per fact; drop superseded versions

The four core techniques: write, select, compress, isolate

Write: keep state outside the window

Agents save notes, plans and findings to files, a database or a state object, then read them back later. Anthropic calls this structured note-taking (a to-do list or a NOTES.md file) and cites it as a way to keep progress across context resets. LangChain lists scratchpads and long-term memories under the same heading.

Select: bring in only what this step needs

Selection covers classic retrieval-augmented generation (embed, search, insert the top results), always-loaded instruction files, memory lookups and even retrieval over tool descriptions so that only relevant tools are offered. Anthropic describes a shift towards "just in time" retrieval, where the agent holds lightweight references (file paths, queries, links) and loads data with tools at runtime, and notes that a hybrid of some preloaded context plus on-demand exploration often works best. For how agent-driven retrieval differs from a fixed pipeline, see Agentic RAG vs Classic RAG.

Compress: summarise or trim

Compaction summarises a conversation nearing the limit and starts a fresh window from the summary. Anthropic advises tuning summaries for recall first, then precision, and calls clearing old tool results the lightest form of compaction. Trimming simply drops older messages or tool output.

Isolate: split the work

Sub-agents run focused tasks in their own clean windows and hand back a condensed result. Anthropic reports that each sub-agent may use tens of thousands of tokens but typically returns a summary of 1,000 to 2,000 tokens to the lead agent. Sandboxed code execution is another form of isolation: large outputs stay in the sandbox and only the relevant result enters context.

Technique	Best fit (per Anthropic's guidance)	Main trade-off
Compaction	Long back-and-forth conversations	Detail lost in the summary cannot be recovered without the original
Structured note-taking	Iterative work with clear milestones	Notes can go stale or carry errors forward
Sub-agents	Parallel research and exploration	More total tokens and coordination overhead
Just-in-time retrieval	Large corpora, codebases, changing data	Slower; depends on good tools and guidance
Preloaded context	Small, stable reference material	Uses window space on every call

How do the major APIs support context management?

Several providers now offer server-side features that edit or condense context for you. The table records only what each provider's documentation states. As of 15 November 2026.

Field	Anthropic (Claude API)	Google (Gemini API)	OpenAI (Responses API)
Server-side compaction	Yes, beta: compaction on demand or at a token threshold	Not stated by Google in the long-context or caching guides	Yes: `context_management` with `compact_threshold`, plus a standalone `/responses/compact` endpoint; compacted item is encrypted and opaque
Selective clearing	Context editing (beta, header `context-management-2025-06-27`): tool result clearing, default trigger 100,000 input tokens, keeps last 3 tool uses; thinking block clearing	Not stated by Google	Not stated by OpenAI in the conversation state guide
Server-held conversation state	Not stated by Anthropic for the Messages API (client resends history)	Stateful interactions via `previous_interaction_id`, referenced in the caching guide	Conversations API and `previous_response_id`; responses stored 30 days by default unless `store: false`
Memory feature	Memory tool `memory_20250818`: client-side file store under `/memories`	Not stated by Google in the guides checked	Not stated by OpenAI in the guides checked
Prompt or context caching	Yes (see Model Pricing Explained)	Implicit caching on by default for Gemini 2.5 and newer; minimum 4,096 tokens on Gemini 4 models, 2,048 on Gemini 2.5	Yes (see Model Pricing Explained)
Billing note	Clearing invalidates cached prefixes; `clear_at_least` exists to make clearing worth the cache loss	For Gemini 4 agent loops, the previous request is cached only if the new request adds at least 2,050 tokens	Earlier input tokens in a `previous_response_id` chain are still billed as input

"Not stated" means the documentation pages checked did not describe the feature; it does not mean the provider cannot support the pattern in your own code. All three patterns (summarise, trim, store notes) can be built client-side on any model. Caching discounts and minimums are covered in Model Pricing Explained.

Where memory fits

Memory is context that outlives a single window. In practice it is a store (files, a database or a vector index) plus rules for what gets written, when it is read back and when it expires. Anthropic's memory tool is a documented example of the client-side model: the model only requests operations (view, create, str_replace, insert, delete, rename) and the application executes them against storage it controls. Its documentation tells implementers to validate every path to block traversal outside /memories, cap file sizes and expire files that have not been used. Context editing can warn the model as it nears the clearing threshold so it can save important facts to memory before old tool results are removed. For memory types and architectures in more depth, see AI Agent Memory Explained.

Context engineering and caching: ordering matters

Prompt caching rewards stable prefixes. Google's caching guide advises putting large, common content at the start of the prompt and sending similar-prefix requests close together. This creates a practical layout rule:

Stable first: system prompt, tool definitions, long reference documents.
Slowly changing next: memory summaries, session-level context.
Volatile last: recent turns, fresh tool results and the current question.

Every edit to an early part of the prompt invalidates the cache from that point. That is why Anthropic's docs flag that clearing tool results or thinking blocks breaks cached prefixes, and why reordering tools between calls can raise cost without changing behaviour.

How to build a context strategy step by step

Measure first. Log input tokens per call, split by component (system, tools, retrieved data, history). Most providers return cached and uncached input counts in the usage object.
Write a minimal system prompt. Run it on your target model and add instructions only for failures you observe.
Prune the tool set. Remove overlapping tools, shorten descriptions and make tools return compact, structured output rather than raw dumps.
Choose retrieval style. Pre-retrieve for small, stable corpora; give the agent search and read tools for large or changing ones.
Add a long-task strategy. Pick compaction, note-taking or sub-agents based on task shape (see the technique table), and set thresholds well below the hard limit.
Lay out for caching. Order content stable to volatile and keep the prefix byte-identical across calls.
Evaluate. Build a fixed set of long tasks and compare success rate, cost and latency before and after each change. Keep the uncompacted transcript for debugging.
Roll back cleanly. Keep context settings (thresholds, keep counts, tool lists) in configuration so a regression can be reverted without a code release.

What should you measure?

Input tokens per successful task: the clearest signal that context is lean or bloated.
Cache hit ratio: cached input tokens divided by total input tokens.
Task success on long runs: success rate on your fixed long-task set, before and after compaction or clearing changes.
Compaction frequency and loss: how often compaction fires and how often a task fails after it because a needed detail was dropped.
Tool result share: percentage of the window taken by tool output; a rising share usually means tools return too much.
Time to first token: latency grows with input length, so track it alongside size.

Where security fits

Every component in the window is something the model may act on. Retrieved web pages, documents, tool results and memory entries can all carry instructions written by someone other than the user, which is the root of indirect prompt injection (OWASP LLM01:2025 Prompt Injection). Memory makes this persistent: a poisoned entry is re-read in later sessions. Secrets placed in context can also be echoed back or logged. Practical consequences for context design:

Label provenance: keep trusted instructions separate from untrusted data in the prompt structure, and record where each retrieved chunk came from.
Validate before writing memory: treat memory writes as data changes that need checks, not as free notes.
Keep credentials out of context: broker tool credentials outside the model rather than pasting keys into prompts.
Enforce permissions outside the model: retrieval should honour the user's access rights before content reaches the window.

The security side is covered in depth in the AI Agent Harness and Context Engineering Security guide, the Context Window Data Exposure guide, the Prompt Injection Guide and the Permission-Aware RAG Guide.

Practitioner checklist

Log input tokens per call by component and review the largest contributors.
Keep the system prompt minimal and structured, and change it only in response to observed failures.
Remove overlapping tools and make every tool return compact output.
Choose compaction, note-taking or sub-agents deliberately for each long-running task type.
Order prompts stable to volatile and keep the cached prefix identical between calls.
Check each provider's documented context management features and their beta status before relying on them.
Treat retrieved content, tool output and memory as untrusted input, and validate memory writes.
Evaluate changes on a fixed set of long tasks, tracking success, cost and latency together.

Context Engineering Explained: How Agents Choose What the Model Sees