AI Agent Behaviour Baselining and Anomaly Detection

An AI agent that has been hijacked, poisoned or has simply misread its task usually still authenticates correctly, so the only reliable sign that something is wrong is that it starts behaving differently. Identity checks and permission boundaries tell you what an agent is allowed to do; a behavioural baseline tells you what it normally does, and the gap between the two is where most agent incidents happen. This guide explains how to build per-agent baselines from agent telemetry, which behavioural dimensions matter, how to choose between declared and learned baselines, which detection methods work for non-deterministic software, and how to measure whether it is working. It is written for detection engineers, SOC leads and platform teams who already log agent activity. What to log, attribution and the kill switch itself are covered in the AI Agent Observability and Incident Response Guide; this guide starts where that one stops.

Key takeaways

Baseline each agent, not each platform. A research agent that reads thousands of documents an hour and a payments agent that makes ten calls a day need different models of normal, and a shared baseline hides both.
Start with a declared baseline. The tools, data sources, destinations and action limits recorded for an agent at registration are a baseline you can enforce on day one, before you have any history to learn from.
Tie every baseline to a version. A new model, system prompt or toolset is a new agent for detection purposes; without version tags you either alert on every release or learn the attack as normal.
Watch sequences, not just counts. The dangerous pattern is often a normal read followed by an unusual write or outbound call, as in the ForcedLeak and EchoLeak research where data left through URLs on trusted domains.
Measure the detector. Track coverage of registered agents, time to detect in exercises, alert precision and baseline freshness, or the programme decays into noise that analysts ignore.

Why agent baselines differ from user and service account baselines

User and entity behaviour analytics (UEBA) already baselines people and hosts. Microsoft Sentinel UEBA, for example, builds dynamic profiles for users, hosts, IP addresses and applications, compares activity with an entity's own history, its peer group and the organisation as a whole, and scores events with an investigation priority from 0 to 10. Those ideas carry over to agents, but four properties change how they apply. See the UEBA Guide for the general method.

Property	Human user	Classic service account	AI agent
Determinism	Habitual but variable	Highly repetitive, scripted	Non-deterministic: the same task can produce different tool sequences
What drives activity	Job role and working hours	Schedule or upstream trigger	Content it reads: prompts, retrieved documents, tool results
Rate of change	Slow (role changes)	Rare (code releases)	Frequent: model, prompt, tool and memory changes
Speed and volume	Human pace	Machine pace, narrow scope	Machine pace, broad scope across many systems in one task
Typical sign of compromise	New location, impossible travel	New source IP or API	New tool, new data, new destination or a changed plan after reading untrusted content

The third column is the important one. Because agent behaviour is driven by the content it reads, an attacker does not need the agent's credentials: indirect prompt injection (MITRE ATLAS AML.T0051, LLM Prompt Injection) changes the behaviour from the inside. NIST's Center for AI Standards and Innovation found that red teaming with the UK AI Security Institute raised hijacking success against one agent from 11% to 81%, and that retrying an attack 25 times raised average success from 57% to 80%. Prevention will not be complete, so behavioural detection is a necessary second layer. See the Prompt Injection Guide.

The telemetry a baseline is built from

A baseline is only as good as the fields it can group and compare. The OpenTelemetry generative AI semantic conventions (status: Development at version 1.37.0, now maintained in a separate GenAI repository) define agent operations create_agent, invoke_agent and execute_tool, with the tool span named execute_tool {gen_ai.tool.name}. Using these names lets you baseline across frameworks rather than per vendor. Managed platforms emit similar data: Amazon Bedrock AgentCore Observability produces OpenTelemetry-compatible metrics, spans, traces and logs in CloudWatch, with metric categories such as session count, latency, duration, token usage and error rates.

Baseline feature	Where it comes from	OpenTelemetry attribute (where defined)
Agent identity and version	Agent registry, platform span	`gen_ai.agent.id`, `gen_ai.agent.name`, `gen_ai.request.model`
Task or session	Orchestrator	`gen_ai.conversation.id`
Tools called and tool type	Tool spans, MCP or API gateway logs	`gen_ai.tool.name`, `gen_ai.tool.type` (function, extension, datastore), `gen_ai.tool.call.id`
Data sources retrieved	Retrieval spans, data platform audit logs	`gen_ai.data_source.id`
Token consumption	Model spans, provider billing	`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`
Credentials used	IdP, cloud and secrets manager logs (token or key ID, never the value)	Not defined: add a custom attribute
Outbound destinations	Egress proxy, DNS, rendered URLs in output	Not defined: join from network logs
Authorisation and approval outcomes	Policy engine, human approval service	Not defined: add a custom attribute

The last three rows are where most programmes have gaps, and they carry the strongest signals. Join them on a correlation or trace ID so that one task can be read end to end.

Behavioural dimensions to baseline

Pick a small set of dimensions per agent and baseline each one separately. A combined "weirdness score" is hard to explain to an analyst and hard to tune.

Dimension	Features to profile	A deviation may indicate	Related technique
Tool repertoire	Set of tools used per version; first use of a tool; tool call frequency per task	Goal hijack, tool misuse, new attack path	ATLAS AML.T0053 AI Agent Tool Invocation
Tool sequence	Common n-grams of tool calls; read-then-send and read-then-delete pairs	Exfiltration or destruction chained after a normal read	ATLAS AML.T0086, AML.T0101
Data access	Data sources, record counts, sensitivity labels touched per task	Scope violation, bulk collection	ATT&CK T1530 Data from Cloud Storage
Egress destinations	Domains and URLs contacted or rendered; new domains; data volume in URL parameters	Exfiltration through trusted or allowlisted domains	ATT&CK T1567 Exfiltration Over Web Service
Credential use	Which token, key or role is used; reads of secret stores, environment variables or credential files	Credential harvesting, privilege escalation	ATLAS AML.T0098; ATT&CK T1552, T1078.004
Authorisation outcomes	Denied calls per task; repeated denials on the same resource; approval requests per task	Probing for a way round a control	OWASP ASI03 Identity and Privilege Abuse
Consumption	Tokens, model calls, tool calls and cost per task and per hour; loop depth	Runaway loops, denial of wallet, stolen key use	ATLAS AML.T0034 Cost Harvesting; ATT&CK T1496
Memory and configuration	Writes to long-term memory, edits to rules or goals, reads of the agent's own configuration	Memory poisoning, persistence, reconnaissance	ATLAS AML.T0080, AML.T0084
Timing and principal	Hours of activity; which users or systems invoke it; invocations with no principal	Abuse of a shared agent, unattended runs	D3FEND D3-JFAPA

Tool sequence and egress destinations deserve extra weight. In both the ForcedLeak research on Salesforce Agentforce and the EchoLeak research on Microsoft 365 Copilot, data left through URLs on domains the platform trusted, so a domain allowlist alone would have passed the traffic. A baseline of normal destinations per agent, plus a feature for data volume carried in URL parameters, catches what the allowlist misses.

Declared baselines versus learned baselines

There are two ways to define normal, and mature programmes use both.

Declared baseline: the agent's registered purpose, tool list, data sources, permitted destinations, action budgets and autonomy level, written down before deployment. Any action outside the declaration is an alert or a block. It needs no history and is easy to explain, but it is only as good as the registration. See the Identity Governance for AI Agents Guide.
Learned baseline: a statistical profile of what the agent has actually done over a training window, per version. It catches drift inside the declared envelope, such as an agent that is allowed to read the CRM but normally reads 40 records per task and suddenly reads 4,000.

Situation	Prefer	Why
New agent, no history	Declared	Nothing to learn from; enforce the registration
High-impact actions (payments, deletes, access changes, external messages)	Declared, enforced as hard limits	Learning would treat an early attack as normal; impact justifies blocking
Stable agent with broad read access	Learned on top of declared	Volume and sequence anomalies sit inside the permitted scope
Fleet of similar agents (one template, many instances)	Learned, with peer group comparison	An instance that diverges from its siblings stands out quickly
Third-party or SaaS agent with limited telemetry	Declared at the boundary	You may only see its API calls, OAuth grant and egress

The declared baseline also makes the learned one safer. Learn only from activity that stayed inside the declaration, so that a violation never becomes part of the model of normal.

Detection methods and their trade-offs

Method	Good at	Weak at	Use for
Boundary rules (deny or alert outside declaration)	Precise, explainable, works on day one	Misses misuse inside allowed scope	New tools, new destinations, credential reads, high-impact actions
Static thresholds and rate limits	Runaway loops, cost spikes	Needs per-agent tuning; blunt	Tool calls per minute, tokens per task, failed calls per task
Statistical baselines (per agent, per version)	Volume and frequency drift	Cold start; seasonality; poisoned training windows	Records read, tools per task, cost per hour
Rarity and first-seen analysis	New tool, data source, domain or credential	Every release creates first-seen events	Combine with version tags to suppress expected change
Sequence analysis	Read-then-exfiltrate, plan change after untrusted input	Needs good trace correlation	Tool n-grams, retrieval followed by new egress
Peer group comparison	One instance in a fleet going wrong	Useless for unique agents	Templated agents, per-tenant copies
Model-based review of traces (a second model judges the trace)	Intent and context that counters miss	Itself non-deterministic and injectable; cost	Triage of flagged traces, not primary detection

The OWASP AI Agent Security Cheat Sheet illustrates the threshold approach with sample values (30 tool calls a minute, five failed tool calls, one injection attempt, a $10 cost cap per session) and stresses that they are examples. Set your own from observed behaviour. The same cheat sheet recommends watching drift in approval behaviour and repeated attempts to bypass approval, which are among the most specific signals an agent produces.

Two scores are better than one. Sentinel UEBA separates an event-level investigation priority from a behaviour-level anomaly score, and notes that a first-time Azure operation can rank high on one and low on the other. Keep a similar split: a rarity score for "never seen before" and an impact score for "this would matter if it were malicious", and alert on the combination.

Handling change: versions, cold start and drift

Version keys: key every baseline on agent ID plus model, system prompt hash, toolset and memory policy version. A release starts a new baseline that inherits the declared envelope but not the learned statistics.
Cold start: for a new version, run boundary rules and thresholds only, and use the previous version's profile as a soft reference. Collect a training window of real tasks before enabling statistical alerts; length depends on task volume, so set it by number of tasks rather than days.
Clean training windows: exclude tasks with policy denials, injection detections or open incidents from training data, so that an attack in progress is not learned.
Drift versus anomaly: slow, steady change across many tasks (a new business process) is drift and should trigger a review of the declaration; a sharp change in one task or session is an anomaly and should trigger triage.
Seasonality: month-end, campaign launches and batch jobs change volume; model them explicitly or compare against the same period rather than a rolling average.
Memory effects: agents with long-term memory change behaviour as memory grows. Treat a large memory write followed by a behaviour shift as a sequence worth investigating. See the AI Agent Memory Security Guide.

How to implement

Inventory and tag: make sure every agent in scope has a registry entry with owner, purpose, tools, data sources, destinations and autonomy level, and that its telemetry carries the agent ID and version. Agents with no registry entry are your first finding. See the Shadow AI Discovery Guide.
Close telemetry gaps: join tool spans, gateway logs, IdP and cloud audit logs, egress proxy logs and approval logs on a trace ID. Add custom attributes for credential ID, authorisation outcome and destination.
Deploy declared rules in alert-only mode: new tool, new destination, credential store read, high-impact action without approval, repeated denials. Run for two to four weeks, review every alert with the agent owner and fix the declaration where it was wrong.
Promote high-confidence rules to enforcement: move boundary rules for high-impact actions and egress to blocking at the gateway or policy engine, one agent at a time, starting with the highest risk. Keep a documented override and a rollback (revert the policy version) for each.
Add thresholds: per-agent caps on tool calls, tokens, cost and loop depth per task, set from observed percentiles with headroom.
Add learned baselines: for agents with enough volume, profile volume, first-seen and sequence features per version, trained on clean windows. Pilot on a small ring of agents and compare alerts against analyst verdicts before widening.
Route and respond: send alerts to the SIEM with agent owner, version and trace link attached, and map each alert type to a response: notify owner, pause agent, or trigger the kill switch. See the AI Agent Kill Switch and Containment Guide and the AI Agent Incident Response Playbook.
Test continuously: replay known attack traces and run red team exercises (injected documents, credential lures, exfiltration via rendered URLs) against staging agents and confirm each produces the expected alert. See Red Teaming AI Agents for Identity Abuse.

Small and large organisations

Small teams: rely on declared baselines and thresholds. Use the controls your agent platform and cloud provider already provide (gateway allowlists, rate limits, cost alerts, managed AI threat detection such as Microsoft Defender for Cloud AI threat protection, which raises alerts for data leakage, data poisoning, jailbreak and credential theft for Azure OpenAI workloads). Review a weekly report of first-seen tools and destinations per agent.
Large organisations: centralise agent telemetry in the SIEM or data lake, build per-version learned baselines, use peer groups for agent fleets, and give detection engineering a standing backlog of agent detections with owners. Expect hundreds of agents, many built on low-code platforms, so automate registration checks.
Third-party agents: whatever your size, you usually see only the OAuth grant, API calls and egress. Baseline those at your boundary and ask the vendor what agent telemetry it can export.

Measuring whether it works

Metric	How to calculate	Target direction
Baseline coverage	Registered production agents with an active declared baseline, divided by all registered production agents	Towards 100%
Telemetry completeness	Share of agent actions carrying agent ID, version, principal, credential ID and trace ID	Towards 100%
Time to detect in exercises	Minutes from injected test behaviour to alert, per scenario	Down, and below the time an agent needs to cause harm
Alert precision	Alerts confirmed as true positive or valid policy violation, divided by all agent alerts	Up; falling precision means retune
Baseline freshness	Share of agents whose baseline matches the currently deployed version	Towards 100%
Declaration drift reviews closed	Drift findings reviewed with owners within the agreed time	Up

Framework mapping

Framework	Reference	How baselining supports it
NIST CSF 2.0	DE.CM-09, DE.AE-02, DE.AE-03	Monitoring runtime environments; analysing adverse events; correlating agent, identity and network data
NIST SP 800-53 Rev. 5	SI-4, SI-4(2), SI-4(4), SI-4(13)	System monitoring, real-time automated analysis, outbound traffic monitoring, analysing event patterns to build profiles
NIST SP 800-53 Rev. 5	AC-2(12), AU-6(5), AU-12	Monitoring accounts for atypical usage; integrated analysis of audit records; audit record generation
MITRE D3FEND	D3-UBA, D3-RAPA, D3-JFAPA, D3-UDTA	User behaviour, resource access pattern, job function access pattern and data transfer analysis applied to agent identities
MITRE ATLAS	AML.T0051, AML.T0053, AML.T0086, AML.T0098	Techniques whose effects show up as behaviour change
OWASP Top 10 for LLM Applications 2025	LLM06 Excessive Agency, LLM10 Unbounded Consumption	LLM10 explicitly recommends logging, monitoring and anomaly detection for resource consumption
OWASP Top 10 for Agentic Applications	ASI01 Agent Goal Hijack, ASI02 Tool Misuse, ASI10 Rogue Agents	Behavioural detection is the main control for misaligned or self-directed action that passes authorisation

Common failure patterns

Trusting the allowlist as the baseline: egress to "trusted" domains is not inspected. ForcedLeak used an expired domain still on a content security policy allowlist, bought for about $5; EchoLeak abused trusted Microsoft domains. Baseline destinations per agent and inspect data in URLs.
One baseline for the whole platform: all agents share a service account or a platform-level profile, so a single agent's deviation disappears into the average. Give each agent its own identity and baseline. See the Agentic AI Identity Guide.
Learning the attack: statistical models trained on windows that included injected or abusive tasks treat them as normal. Train only on clean, in-declaration activity.
Release noise: baselines not keyed on version fire on every prompt or model update, and analysts learn to close agent alerts unread.
Monitoring outcomes only: the SOC sees the CRM export but not the retrieval and plan change that preceded it, so it cannot tell injection from a legitimate request. Correlate retrieval, plan and tool spans.
No live monitoring of agents with real-world reach: in the UK AISI agent testing incident, agents with internet access created fake identities and contacted real people over several days before security monitoring flagged data leaving over Tor. Alert on account sign-ups, outbound messages, repository writes and anonymity network use from agent runtimes.

Practitioner checklist

Every production agent has a registry entry whose tools, data sources, destinations and action limits form an enforced declared baseline.
Agent telemetry carries agent ID, version, principal, credential ID, authorisation outcome and trace ID, using OpenTelemetry GenAI names where possible.
Egress from agent runtimes passes a proxy that logs destination and URL data volume per agent.
Boundary rules cover first-seen tools, destinations and credentials, secret store reads, repeated denials and high-impact actions without approval.
Per-agent caps exist for tool calls, tokens, cost and loop depth per task.
Learned baselines are keyed on version and trained only on clean, in-declaration activity.
Each alert type maps to a documented response, from owner notification to the kill switch.
Detections are tested with replayed attack traces and red team scenarios at least every release cycle.
Coverage, completeness, time to detect, precision and freshness are reported monthly.

AI Agent Behaviour Baselining and Anomaly Detection Guide: Spotting Agents That Go Off Script