How should security teams implement just-in-time context in long-running AI agents?

Security teams should keep the agent window small and persistent only for stable instructions, task state, and compact references. Load large policies, tool details, and source data only when a step needs them, then let them fall out of context. This reduces context rot, lowers repeated token cost, and keeps the model focused on current work instead of stale payloads.

Why This Matters for Security Teams

Just-in-time context is a control problem, not only a prompt engineering choice. Long-running agents accumulate stale instructions, exposed secrets, and irrelevant evidence unless teams deliberately separate durable state from ephemeral context. That distinction matters because agentic systems can call tools, take actions, and reuse memory in ways that amplify a small mistake into a broader workflow failure. The current guidance in the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework points toward minimizing unnecessary exposure and documenting where context enters, persists, and expires.

Security teams often get this wrong by treating the agent’s prompt window like a permanent workspace rather than a controlled runtime. That leads to repeated inclusion of broad policy packs, old tickets, and copied credentials that should never have been retained. It also makes audits harder because the team cannot tell whether a decision came from current evidence or from lingering memory. The practical goal is to keep only what the next step needs, while preserving enough state to maintain continuity.

In practice, many security teams encounter context leakage only after an agent has already acted on outdated instructions rather than through intentional context design.

How It Works in Practice

Implementing just-in-time context means designing the agent workflow so information is fetched, used, and discarded at step boundaries. Stable items belong in a small durable state store, such as task identifiers, approved objectives, and compact summaries. Everything else should be injected only when needed, including policy excerpts, tool schemas, source documents, and retrieval snippets. This approach is aligned with OWASP Top 10 for Agentic Applications 2026 guidance on limiting agent overreach and with threat modeling patterns from the CSA MAESTRO agentic AI threat modeling framework.

• Keep a short system policy that defines non-negotiables, then load task-specific instructions only for the active step.
• Use retrieval controls so documents are fetched on demand, not embedded into every turn of the conversation.
• Summarize completed steps into compact state, and expire raw inputs when they are no longer required.
• Tag sensitive material such as secrets, tokens, and certificates so the agent never persists them in memory or logs.
• Record which context items were available at each action for later review and incident response.

For adversarial testing, the MITRE ATLAS adversarial AI threat matrix is useful for thinking about prompt injection, data poisoning, and tool misuse that can exploit oversized context. Teams should also examine whether context fetches are authenticated, authorised, and scoped to the minimum necessary source. These controls tend to break down when agents span many tools and asynchronous steps because stale summaries start to substitute for verified source data.

Common Variations and Edge Cases

Tighter context windows often increase orchestration overhead, requiring organisations to balance lower exposure against more frequent retrieval and summarisation. That tradeoff becomes visible in long-running investigations, customer support workflows, and code-assistance agents where the task naturally spans many turns. Best practice is evolving here, and there is no universal standard for how much state should remain persistent versus ephemeral.

Some environments need broader retention for continuity, but the retention boundary should still be explicit. For example, a case-handling agent may keep a working summary while discarding source transcripts, whereas an engineering agent may retain repository identifiers but not full file contents. If the workflow touches regulated or high-risk decisions, the governance layer should define what context can survive between steps and what must be re-fetched from an approved source. This is especially important when agents operate with human approvals, because approval does not make stale context safe.

Operationally, teams should be cautious with shared memory, cross-session recall, and “helpful” long summaries that quietly reintroduce sensitive or outdated material. In those cases, just-in-time context should be paired with provenance checks, expiry rules, and explicit source-of-truth lookups. The approach is strongest when the environment has well-defined APIs and structured data, and weakest when unstructured documents, ad hoc tool calls, and free-form human instructions all compete for the same context budget.

Related resources from NHI Mgmt Group

• How should security teams implement AI agents in cloud and application security workflows without losing control over context and risk?
• Should security teams require just-in-time access for AI agents?
• How should security teams govern AI agents that use Model Context Protocol?
• How should security teams implement short-lived credentials for AI agents?