Why do AI agents make open-source detection harder to operate?

Why This Matters for Security Teams

Open-source detection stacks work best when the thing being monitored behaves consistently enough to match a rule, a model, or a known attack chain. AI agents disrupt that assumption. Their tool use, prompts, memory, and model updates can change the same workflow from one run to the next, so the signal often looks different even when the intent is similar. That creates drift in detections, brittle exceptions, and alert fatigue.

The operational risk is not limited to missed alerts. Teams can also overfit to yesterday’s behaviour and end up flagging legitimate agent activity as suspicious, which makes security and platform owners less willing to trust the detections. Current guidance from the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10 both point to the same problem: governance has to account for changing model behaviour, not just endpoint telemetry.

In practice, many security teams discover the detection gap only after an agent has already touched data, called tools, or triggered a noisy incident review, rather than through intentional baseline testing.

How It Works in Practice

Agent detection is harder because the observable behaviour is shaped by multiple moving parts: the model version, the system prompt, the context window, the available tools, the retrieval layer, and any policy guardrails. A rule that works for one agent may fail for another that uses the same model but a different tool chain. That is why practitioners increasingly build detections around behaviours, not just signatures.

Useful monitoring usually combines identity, execution, and content signals. For example, security teams may correlate which agent identity invoked a tool, what resource it reached, what output it produced, and whether the sequence matched expected task intent. The MITRE ATLAS adversarial AI threat matrix is helpful for mapping adversarial patterns such as prompt injection, model manipulation, and abuse of agent tool access. The same logic applies when using the CSA MAESTRO agentic AI threat modeling framework to identify where the agent’s trust boundaries can be crossed.

• Track each agent as a distinct identity with its own approved tools, scope, and risk profile.
• Baseline the normal sequence of actions, not just the final output.
• Correlate prompt, retrieval, tool invocation, and destination system activity.
• Flag high-risk transitions such as privilege escalation, unusual data movement, or external side effects.
• Retest detections after model updates, policy changes, or tool changes.

Detection also improves when teams add output validation and task-aware approvals, especially for actions that can modify records, send messages, or execute code. These controls tend to break down when agents are granted broad tool access without stable ownership, because the same workflow can appear legitimate one moment and anomalous the next.

Common Variations and Edge Cases

Tighter detection usually increases tuning overhead, requiring organisations to balance better visibility against alert volume, maintenance cost, and operational latency. That tradeoff becomes sharper as agents are allowed to act across multiple systems or are reused for different business functions.

Some environments are easier to govern than others. Single-purpose agents with fixed tools can often be monitored with relatively stable baselines, while general-purpose agents that browse, retrieve, write, and execute across many apps create far more variability. There is no universal standard for this yet, but best practice is evolving toward per-agent policy, per-tool logging, and behavioural validation tied to the agent’s approved job role.

Open-source detection also struggles when the underlying model changes frequently or when the organisation relies on third-party orchestration that hides context and tool calls. In those cases, security teams may need to extend their controls with provenance checks, stronger logging at the integration layer, and independent review of high-impact actions. The issue is especially visible when an agent can generate apparently benign intermediate steps that only become risky after a sequence is completed, which makes single-event alerting unreliable. The NIST Cybersecurity Framework 2.0 is useful here because it forces teams to treat detection as an ongoing governance and resilience problem, not a one-time control deployment.

Related resources from NHI Mgmt Group

• Why do AI agents and automated attackers make traditional detection harder?
• Why do AI agents make compromise detection harder than traditional systems?
• Why do AI agents make non-human identity governance harder?
• Why do AI agents make IAM and NHI risk harder to manage?