Security teams buy monitoring and observability tools as if they were the same thing, then discover during an incident that neither can answer the question they now need answered. Monitoring watches for conditions you defined in advance; observability is the property of a system that lets you ask new questions of its telemetry after the fact. The difference decides what you catch, what it costs and what evidence survives. This guide is for detection engineers, SOC leads and platform owners deciding how to split budget and ownership between the two, including for AI agents and LLM applications. It does not repeat pipeline hardening or SRE mechanics, which sit in the Observability, telemetry and SRE for security teams guide.
Key takeaways
- Monitoring answers known questions. It checks predefined signals against thresholds or rules and pages someone; it is only as good as the list of failures you anticipated.
- Observability answers questions you did not plan. It depends on rich, correlated, high-cardinality telemetry (traces, structured logs, metrics with context) that you can slice in new ways after something odd happens.
- Traces are the security gap most teams miss. A trace shows which identity called which service with what outcome across a whole request, which is the path an investigator reconstructs after token misuse or API abuse.
- Cost pushes observability towards sampling, and sampling deletes evidence. Keep errors, authentication paths and privileged actions at 100%, and never let audit logs ride the same sampled pipeline.
- Agents make the split sharper. You cannot write alert rules for every way an agent might misuse a tool, so agent and LLM telemetry must be observable, while a small set of hard conditions stay as monitored alerts.
The short answer
Run monitoring for the failures you can name: an identity provider returning errors, a spike in 401 responses, a detection pipeline falling behind, a disabled log source. Build observability so that when monitoring fires, or when a threat report or a researcher's email arrives, you can explain what actually happened without shipping new instrumentation first. Most organisations need both, owned differently: monitoring is a set of rules with an on-call owner, observability is a data capability with a data owner and a retention budget.
Google's SRE book defines monitoring as collecting, processing, aggregating and displaying real-time quantitative data about a system. The OpenTelemetry project describes observability as letting you understand a system from the outside by asking questions about it without knowing its internals, which it ties to troubleshooting "unknown unknowns". Monitoring is an activity; observability is a property you design in through instrumentation.
Two models side by side
| Dimension | Monitoring | Observability |
|---|---|---|
| Question it answers | Is a known bad condition happening now? | Why is this happening, and what else did this actor touch? |
| Threat model | Known failure modes and known attack patterns: outages, brute force, disabled logging, quota exhaustion | Novel behaviour: a valid token used in an unusual sequence, an agent chaining tools in an unexpected order, a slow exfiltration below every threshold |
| Typical data | Aggregated metrics, health checks, rule matches on logs | Traces with spans, structured logs carrying trace IDs, metrics with dimensions such as tenant, client and route |
| Architecture in words | Agents or exporters emit metrics; a time-series store evaluates rules; an alert manager routes pages; dashboards summarise | Instrumentation emits correlated signals; a collector pipeline enriches, redacts and samples; a query engine lets analysts pivot across signals by any attribute |
| Cardinality | Low by design: a handful of labels keeps storage and alert evaluation cheap | High by design: user, client ID, token ID hash, route, tenant, model, tool name |
| Main failure mode | Silent on anything not anticipated; alert fatigue when rules are vague | Cost blow-out, sampled-away evidence, sensitive data captured in attributes |
| Output | Pages, tickets, dashboards | Investigations, new detection rules, root cause, evidence |
| Owner | On-call team for the service or the SOC | Platform or data team, with security as a major consumer |
Visibility is the third word that gets mixed in. Visibility usually means coverage: do you collect anything at all from an asset, account or network segment. You can have visibility without observability (a firewall log that tells you a connection happened but not which workload identity opened it) and monitoring without visibility (alerting perfectly on the half of the estate you collect from).
Telemetry types and what each is good for
| Signal | Monitoring use | Observability use | Security caveat |
|---|---|---|---|
| Metrics | Golden signals: latency, traffic, errors, saturation; thresholds and burn rates | Dimensional breakdowns, for example 403 rate by client_id and route | High-cardinality labels are the first thing cut to save money, and they are often the ones an investigator needs |
| Logs | Rule matches: failed logins, policy changes, log source stopped | Context for a span when they carry trace and span IDs | Logs often capture tokens, session IDs and personal data; the OWASP Logging Cheat Sheet lists access tokens and session IDs as data to exclude |
| Traces | Rarely used for alerts beyond error rates per service | Full request path across gateway, services, queues and data stores with identity and outcome per hop | Commonly sampled; traceparent headers can be spoofed from outside unless you restart traces at the edge |
| Audit logs | Alert on high-risk events such as role grants and key deletion | System of record for who did what | Must be complete and tamper-evident: never sampled, never dropped to save cost |
| Health checks and synthetic probes | Black-box monitoring of what a user sees | Limited | Probe credentials are non-human identities and need the same governance as any other |
The SRE book's split between white-box monitoring (internal metrics, logs, handlers) and black-box monitoring (testing behaviour as a user would see it) maps neatly to security. Black-box checks catch an identity provider that is down; only white-box telemetry shows that it is up but issuing tokens to a client that should have been disabled.
What each catches, and what each misses
| Scenario | Monitoring | Observability |
|---|---|---|
| Password spraying against a login endpoint | Catches it: failed authentication rate by source crosses a threshold | Adds which accounts succeeded afterwards and what those sessions did |
| Stolen but valid OAuth token replayed from a new network | Usually misses it: every request returns 200 | Catches it when traces carry client ID, token identifier hash and source, so you can pivot from one odd request to everything the token did |
| Detection pipeline 40 minutes behind | Catches it if you monitor freshness as an SLI | Explains which stage is saturated |
| Logging disabled on a host or tenant (ATT&CK T1685 Disable or Modify Tools) | Catches it with a "source went silent" rule | Cannot help after the fact: the data is gone |
| Agent calls an export tool it has never used before | Misses it unless someone wrote that exact rule | Catches it when tool spans carry the tool name, agent identity and delegating user |
| Slow data exfiltration under every volume threshold | Misses it | Finds it in a hunt by summing response size per identity per day |
The pattern is consistent. Monitoring is strong on availability and loud attacks, and on the absence of data. Observability is strong on misuse of valid credentials and on behaviour nobody predicted, but only if the data was captured with identity context and kept long enough.
Security use of traces
Distributed tracing links spans (single units of work with timing and attributes) into one trace per request. W3C Trace Context, a W3C Recommendation since 2020 with editorial updates published on 23 November 2021, carries that link between services in the traceparent header: a version, a 16-byte trace ID, an 8-byte parent ID and trace flags whose lowest bit is the sampled flag. For security, three practices make traces useful rather than risky:
- Put identity on spans, not secrets: record the authenticated subject, client ID, workload identity and a hash or jti of the token as attributes at the gateway and at each service that makes an authorisation decision. Never record the token itself. See the JWT and Session Tokens in Code and Logs Guide.
- Restart traces at trust boundaries: the specification notes that services may restart traces at trust boundaries to reduce correlation risk, and warns that naively honouring a sampled flag from public callers can overwhelm an application, allow trace ID collisions or inflate cost. Treat inbound trace context from the internet as untrusted input.
- Keep PII out of propagation headers: the specification says vendors must not put personally identifiable or sensitive information in traceparent or tracestate. Their only purpose is correlation.
Traces also help with non-human identities. A service account that normally calls two internal APIs and suddenly fans out to twenty is invisible to a per-service error monitor and obvious in a service graph built from traces. See the Detection Engineering for Non-Human Identities Guide.
Cost, sampling and retention
Observability data grows with traffic and cardinality, so cost is the constraint that shapes it. The OpenTelemetry sampling guidance says a rate of 1% or lower can represent the rest of high-volume traffic accurately, suggests sampling when you generate around 1,000 or more traces per second, and lists cases where you should not sample, including when regulation prohibits dropping data and you cannot route it to cheaper storage. It also names errors and high latency as common keep criteria. Security adds its own keep criteria.
| Data class | Sampling | Hot retention (example) | Cold retention |
|---|---|---|---|
| Audit logs (identity provider, cloud control plane, secrets manager) | None | 90 days searchable | Per legal and regulatory requirement |
| Traces touching authentication, token issuance, admin and payment routes | None (tail-sampling policy keeps 100%) | 30 days | Optional, summarised |
| Error traces | None | 14 to 30 days | No |
| Routine successful traces | Probabilistic, for example 1 to 10% | 7 days | No |
| Monitoring metrics | Aggregated | 13 months for trend and capacity | Downsampled |
The retention values are illustrations to adapt, not requirements from a standard; regulatory minimums are in the Log Retention Requirements Map. Cost levers that do not hurt security: drop labels nobody queries, aggregate health-check spans, shorten debug-log retention, and keep identity attributes even when other dimensions go. The worst lever is a uniform head-sampling rate across everything, which drops the rare abusive request at the same rate as the routine one.
Agent and LLM observability
AI agents are where monitoring alone breaks down. An agent chooses tools, arguments and sequences at run time, so the space of possible misuse cannot be enumerated as alert rules. The OpenTelemetry GenAI semantic conventions, now maintained in their own repository and marked Development status, define spans for operations such as chat, create_agent, invoke_agent, plan and execute_tool, with attributes including gen_ai.agent.name, gen_ai.conversation.id and gen_ai.usage.input_tokens. Prompt and response content attributes (gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions and gen_ai.tool.definitions) are Opt-In, and the conventions warn they may contain sensitive information.
| Need | Monitor (alert rule) | Observe (queryable telemetry) |
|---|---|---|
| Cost and abuse | Token usage per agent above budget | Which conversations and tools drove the spend |
| Tool misuse | Any call to a tool on a deny list; any write tool called without approval | Full tool sequence per conversation with the delegating user and agent identity |
| Prompt injection | Guardrail block rate spike | Which retrieved document or tool output preceded an unexpected action |
| Identity | Agent credential used outside its normal host or hours | Every downstream API call made on behalf of one user across agents |
Two cautions. First, opt-in content capture turns an observability backend into a store of prompts, customer data and possibly secrets, so it needs the same access control and redaction as any sensitive data store; see the LLM Observability Guide and the OpenTelemetry GenAI Semantic Conventions for Security Teams. Second, agent observability is not an audit trail: sampled, mutable traces cannot prove what an agent did. Keep a separate, complete record of tool actions with side effects, as set out in Agent Observability vs Audit Logging.
Worked scenario: one alert, two capabilities
A monitoring rule fires at 02:10: the export API's 200-response volume for one partner client is three times its weekly baseline. That is all monitoring can say. With observability in place, the analyst pivots on client_id in traces and finds that every request carries the same token jti, originates from a cloud network the partner never uses, and calls a bulk endpoint the partner has not called in 90 days. Pivoting on the token hash shows the same token used two days earlier at low volume, below every threshold. The trace attributes identify the issuing grant, so the team revokes the refresh token, disables the client and has a scoped list of exported records for the notification decision.
Without trace-level identity, the team would know only that volume rose. Without the volume monitor, the observable data would have sat unqueried. The Wyndham hardcoded app password breach shows the other side: when a service credential that can read customer login tokens leaks, the first question is whether it was used, and only logged, attributable access can answer it.
Decision guidance
- Start with monitoring when: you have no on-call alerts for identity provider health, log source silence, detection pipeline freshness or authentication failure spikes. These are cheap, well understood and catch the loud failures.
- Invest in observability when: investigations regularly stall at "we cannot tell which identity did that", your estate is microservices or APIs where one request crosses many systems, or you run agents.
- Small organisations: use the observability features of your cloud provider or existing APM tool, add identity attributes at the gateway, and keep audit logs in the SIEM. Do not build a separate trace platform for security.
- Large organisations: run one telemetry pipeline (for example OpenTelemetry Collectors) feeding both observability and security backends, with security-owned tail-sampling and redaction policies, rather than parallel agents on every host.
- Regulated environments: confirm that sampled or short-retention data is never the only copy of anything a regulator or auditor may ask for.
How to implement
- Inventory questions, not tools: list the ten questions your last three investigations could not answer quickly. Each one becomes either a monitoring rule (known condition) or an instrumentation requirement (attribute to capture).
- Set the monitoring baseline: golden-signal alerts for identity, secrets and detection services; "source went silent" rules for every critical log source; freshness SLIs for the SIEM. Follow the SRE rule that every page must be actionable.
- Define the identity attribute standard: subject, client ID, workload identity, token identifier hash, tenant and decision outcome, emitted at the gateway and at authorisation points. Test that no raw token, cookie or password appears.
- Pilot on one high-value path: instrument login, token issuance and one sensitive API end to end. Run a tabletop that answers the scenario above using only the new telemetry.
- Write security sampling policy: tail-sampling rules that keep errors, authentication, privileged and agent tool-call traces at 100%; route the rest to cheaper storage where budget allows.
- Turn findings into rules: every investigation that used observability should end with a question: can this become a monitored condition? This is how observability feeds monitoring. See the Detection Engineering Guide.
- Roll back safely: if instrumentation causes latency or captures sensitive content, disable the specific attributes or sampling rule through collector configuration rather than removing SDKs, and keep a version history of pipeline configuration.
Detection and measurement
- Time to answer: median minutes to answer "what else did this identity do in the last 7 days" during exercises. Target a steady fall.
- Identity coverage of traces: share of traces on sensitive routes that carry subject and client ID attributes. Target above 95%.
- Silent source detection time: minutes between a critical log source stopping and an alert. Target under 30.
- Actionable page rate: share of security pages that led to an action. Low rates mean vague rules; see the Alert Fatigue Reduction Guide.
- Observability to monitoring conversion: number of new monitored rules created from investigations per quarter.
- Sensitive data leakage: count of seeded test secrets found in the observability backend. Target zero.
Framework mapping
| Framework | Reference | Monitoring | Observability |
|---|---|---|---|
| NIST CSF 2.0 | DE.CM-01, DE.CM-09 | Networks, hardware, software and runtime environments monitored for adverse events | Telemetry depth that makes monitoring meaningful |
| NIST CSF 2.0 | DE.AE-02, DE.AE-03 | Alert triggers analysis | Analysis and correlation from multiple sources |
| NIST SP 800-53 Rev 5 | SI-4, CA-7 | System Monitoring; Continuous Monitoring | Supports both |
| NIST SP 800-53 Rev 5 | AU-6, AU-9, AU-11, AU-12 | Audit record review and reporting | Generation, protection and retention of the records investigations depend on |
| CIS Controls v8.1 | 8.2, 8.11, 13.1 | Collect audit logs, review them, centralise security event alerting | Detailed logs and correlation feed reviews |
| ISO/IEC 27001:2022 | Annex A 8.15, 8.16 | 8.16 Monitoring activities | 8.15 Logging |
| MITRE ATT&CK v20 | T1685 Disable or Modify Tools | Alert on telemetry loss | Not applicable once data is lost |
NIST SP 800-137 frames continuous monitoring as visibility into assets, threats and the effectiveness of controls, which is the monitoring half. The draft NIST SP 800-92 Rev. 1 (initial public draft, October 2023) treats log management as generating, transmitting, storing, accessing and disposing of log data, which is the foundation both halves rest on.
Common failure patterns
- Calling dashboards observability: a wall of predefined charts is monitoring. If you cannot ask a new question across signals by an arbitrary attribute, you do not have observability.
- Sampling security evidence away: a uniform 1% head-sampling rate set by the platform team for cost, never reviewed by security, discovered during the first investigation that needed traces.
- No identity on telemetry: traces show service-to-service calls but not which user, client or workload identity drove them, so the trace stops being evidence.
- Secrets in attributes: HTTP header capture or opt-in prompt capture switched on in production, putting bearer tokens and customer data into a platform with broad read access. See the observability platform compromise playbook.
- Not monitoring the monitors: no alert when a log source, collector or agent stops sending, so an attacker who disables tooling gets silence instead of a page.
- Treating traces as audit: relying on observability data for compliance evidence when it is sampled, mutable and short-lived.
Practitioner checklist
- List the questions your last investigations could not answer, and assign each to monitoring or instrumentation.
- Alert on silence for every critical log source, collector and agent.
- Monitor freshness and completeness of the SIEM and detection pipeline as SLIs.
- Add subject, client ID, workload identity and token identifier hash to spans at gateways and authorisation points.
- Restart trace context from the internet at the edge and rate-limit honouring of the sampled flag.
- Write tail-sampling rules that keep errors, authentication, privileged and agent tool-call traces at 100%.
- Keep audit logs on a separate, unsampled, tamper-evident path.
- Keep GenAI content capture off by default; enable it per use case with redaction and access control.
- Seed test secrets and confirm none reach the observability backend.
- Review each investigation for a rule that can move from observability into monitoring.
Standards and references
- W3C: Trace Context
- NIST: SP 800-137 Information Security Continuous Monitoring
- NIST: SP 800-92 Rev. 1 (Draft) Cybersecurity Log Management Planning Guide
- OpenTelemetry: Semantic Conventions for Generative AI
- OpenTelemetry: Observability primer
- OpenTelemetry: Sampling
- Google: Site Reliability Engineering, Monitoring Distributed Systems
- OWASP: Logging Cheat Sheet
Related NHI Mgmt Group resources: Observability, telemetry and SRE for security teams · SIEM Guide · LLM Observability Guide · Agent Observability vs Audit Logging · Logging and Monitoring Policy Template