Join our Newsletter — 33% off our NHI Course
Home› Guides› Monitoring vs Observability: what each catches and where…
Guide SOC & Detection Engineering

Monitoring vs Observability: what each catches and where each goes blind

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 14 November 2026 17 min read
On this page

Security teams buy monitoring and observability tools as if they were the same thing, then discover during an incident that neither can answer the question they now need answered. Monitoring watches for conditions you defined in advance; observability is the property of a system that lets you ask new questions of its telemetry after the fact. The difference decides what you catch, what it costs and what evidence survives. This guide is for detection engineers, SOC leads and platform owners deciding how to split budget and ownership between the two, including for AI agents and LLM applications. It does not repeat pipeline hardening or SRE mechanics, which sit in the Observability, telemetry and SRE for security teams guide.

Key takeaways

  • Monitoring answers known questions. It checks predefined signals against thresholds or rules and pages someone; it is only as good as the list of failures you anticipated.
  • Observability answers questions you did not plan. It depends on rich, correlated, high-cardinality telemetry (traces, structured logs, metrics with context) that you can slice in new ways after something odd happens.
  • Traces are the security gap most teams miss. A trace shows which identity called which service with what outcome across a whole request, which is the path an investigator reconstructs after token misuse or API abuse.
  • Cost pushes observability towards sampling, and sampling deletes evidence. Keep errors, authentication paths and privileged actions at 100%, and never let audit logs ride the same sampled pipeline.
  • Agents make the split sharper. You cannot write alert rules for every way an agent might misuse a tool, so agent and LLM telemetry must be observable, while a small set of hard conditions stay as monitored alerts.

The short answer

Run monitoring for the failures you can name: an identity provider returning errors, a spike in 401 responses, a detection pipeline falling behind, a disabled log source. Build observability so that when monitoring fires, or when a threat report or a researcher's email arrives, you can explain what actually happened without shipping new instrumentation first. Most organisations need both, owned differently: monitoring is a set of rules with an on-call owner, observability is a data capability with a data owner and a retention budget.

Google's SRE book defines monitoring as collecting, processing, aggregating and displaying real-time quantitative data about a system. The OpenTelemetry project describes observability as letting you understand a system from the outside by asking questions about it without knowing its internals, which it ties to troubleshooting "unknown unknowns". Monitoring is an activity; observability is a property you design in through instrumentation.

Two models side by side

DimensionMonitoringObservability
Question it answersIs a known bad condition happening now?Why is this happening, and what else did this actor touch?
Threat modelKnown failure modes and known attack patterns: outages, brute force, disabled logging, quota exhaustionNovel behaviour: a valid token used in an unusual sequence, an agent chaining tools in an unexpected order, a slow exfiltration below every threshold
Typical dataAggregated metrics, health checks, rule matches on logsTraces with spans, structured logs carrying trace IDs, metrics with dimensions such as tenant, client and route
Architecture in wordsAgents or exporters emit metrics; a time-series store evaluates rules; an alert manager routes pages; dashboards summariseInstrumentation emits correlated signals; a collector pipeline enriches, redacts and samples; a query engine lets analysts pivot across signals by any attribute
CardinalityLow by design: a handful of labels keeps storage and alert evaluation cheapHigh by design: user, client ID, token ID hash, route, tenant, model, tool name
Main failure modeSilent on anything not anticipated; alert fatigue when rules are vagueCost blow-out, sampled-away evidence, sensitive data captured in attributes
OutputPages, tickets, dashboardsInvestigations, new detection rules, root cause, evidence
OwnerOn-call team for the service or the SOCPlatform or data team, with security as a major consumer

Visibility is the third word that gets mixed in. Visibility usually means coverage: do you collect anything at all from an asset, account or network segment. You can have visibility without observability (a firewall log that tells you a connection happened but not which workload identity opened it) and monitoring without visibility (alerting perfectly on the half of the estate you collect from).

Telemetry types and what each is good for

SignalMonitoring useObservability useSecurity caveat
MetricsGolden signals: latency, traffic, errors, saturation; thresholds and burn ratesDimensional breakdowns, for example 403 rate by client_id and routeHigh-cardinality labels are the first thing cut to save money, and they are often the ones an investigator needs
LogsRule matches: failed logins, policy changes, log source stoppedContext for a span when they carry trace and span IDsLogs often capture tokens, session IDs and personal data; the OWASP Logging Cheat Sheet lists access tokens and session IDs as data to exclude
TracesRarely used for alerts beyond error rates per serviceFull request path across gateway, services, queues and data stores with identity and outcome per hopCommonly sampled; traceparent headers can be spoofed from outside unless you restart traces at the edge
Audit logsAlert on high-risk events such as role grants and key deletionSystem of record for who did whatMust be complete and tamper-evident: never sampled, never dropped to save cost
Health checks and synthetic probesBlack-box monitoring of what a user seesLimitedProbe credentials are non-human identities and need the same governance as any other

The SRE book's split between white-box monitoring (internal metrics, logs, handlers) and black-box monitoring (testing behaviour as a user would see it) maps neatly to security. Black-box checks catch an identity provider that is down; only white-box telemetry shows that it is up but issuing tokens to a client that should have been disabled.

What each catches, and what each misses

ScenarioMonitoringObservability
Password spraying against a login endpointCatches it: failed authentication rate by source crosses a thresholdAdds which accounts succeeded afterwards and what those sessions did
Stolen but valid OAuth token replayed from a new networkUsually misses it: every request returns 200Catches it when traces carry client ID, token identifier hash and source, so you can pivot from one odd request to everything the token did
Detection pipeline 40 minutes behindCatches it if you monitor freshness as an SLIExplains which stage is saturated
Logging disabled on a host or tenant (ATT&CK T1685 Disable or Modify Tools)Catches it with a "source went silent" ruleCannot help after the fact: the data is gone
Agent calls an export tool it has never used beforeMisses it unless someone wrote that exact ruleCatches it when tool spans carry the tool name, agent identity and delegating user
Slow data exfiltration under every volume thresholdMisses itFinds it in a hunt by summing response size per identity per day

The pattern is consistent. Monitoring is strong on availability and loud attacks, and on the absence of data. Observability is strong on misuse of valid credentials and on behaviour nobody predicted, but only if the data was captured with identity context and kept long enough.

Security use of traces

Distributed tracing links spans (single units of work with timing and attributes) into one trace per request. W3C Trace Context, a W3C Recommendation since 2020 with editorial updates published on 23 November 2021, carries that link between services in the traceparent header: a version, a 16-byte trace ID, an 8-byte parent ID and trace flags whose lowest bit is the sampled flag. For security, three practices make traces useful rather than risky:

  • Put identity on spans, not secrets: record the authenticated subject, client ID, workload identity and a hash or jti of the token as attributes at the gateway and at each service that makes an authorisation decision. Never record the token itself. See the JWT and Session Tokens in Code and Logs Guide.
  • Restart traces at trust boundaries: the specification notes that services may restart traces at trust boundaries to reduce correlation risk, and warns that naively honouring a sampled flag from public callers can overwhelm an application, allow trace ID collisions or inflate cost. Treat inbound trace context from the internet as untrusted input.
  • Keep PII out of propagation headers: the specification says vendors must not put personally identifiable or sensitive information in traceparent or tracestate. Their only purpose is correlation.

Traces also help with non-human identities. A service account that normally calls two internal APIs and suddenly fans out to twenty is invisible to a per-service error monitor and obvious in a service graph built from traces. See the Detection Engineering for Non-Human Identities Guide.

Cost, sampling and retention

Observability data grows with traffic and cardinality, so cost is the constraint that shapes it. The OpenTelemetry sampling guidance says a rate of 1% or lower can represent the rest of high-volume traffic accurately, suggests sampling when you generate around 1,000 or more traces per second, and lists cases where you should not sample, including when regulation prohibits dropping data and you cannot route it to cheaper storage. It also names errors and high latency as common keep criteria. Security adds its own keep criteria.

Data classSamplingHot retention (example)Cold retention
Audit logs (identity provider, cloud control plane, secrets manager)None90 days searchablePer legal and regulatory requirement
Traces touching authentication, token issuance, admin and payment routesNone (tail-sampling policy keeps 100%)30 daysOptional, summarised
Error tracesNone14 to 30 daysNo
Routine successful tracesProbabilistic, for example 1 to 10%7 daysNo
Monitoring metricsAggregated13 months for trend and capacityDownsampled

The retention values are illustrations to adapt, not requirements from a standard; regulatory minimums are in the Log Retention Requirements Map. Cost levers that do not hurt security: drop labels nobody queries, aggregate health-check spans, shorten debug-log retention, and keep identity attributes even when other dimensions go. The worst lever is a uniform head-sampling rate across everything, which drops the rare abusive request at the same rate as the routine one.

Agent and LLM observability

AI agents are where monitoring alone breaks down. An agent chooses tools, arguments and sequences at run time, so the space of possible misuse cannot be enumerated as alert rules. The OpenTelemetry GenAI semantic conventions, now maintained in their own repository and marked Development status, define spans for operations such as chat, create_agent, invoke_agent, plan and execute_tool, with attributes including gen_ai.agent.name, gen_ai.conversation.id and gen_ai.usage.input_tokens. Prompt and response content attributes (gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions and gen_ai.tool.definitions) are Opt-In, and the conventions warn they may contain sensitive information.

NeedMonitor (alert rule)Observe (queryable telemetry)
Cost and abuseToken usage per agent above budgetWhich conversations and tools drove the spend
Tool misuseAny call to a tool on a deny list; any write tool called without approvalFull tool sequence per conversation with the delegating user and agent identity
Prompt injectionGuardrail block rate spikeWhich retrieved document or tool output preceded an unexpected action
IdentityAgent credential used outside its normal host or hoursEvery downstream API call made on behalf of one user across agents

Two cautions. First, opt-in content capture turns an observability backend into a store of prompts, customer data and possibly secrets, so it needs the same access control and redaction as any sensitive data store; see the LLM Observability Guide and the OpenTelemetry GenAI Semantic Conventions for Security Teams. Second, agent observability is not an audit trail: sampled, mutable traces cannot prove what an agent did. Keep a separate, complete record of tool actions with side effects, as set out in Agent Observability vs Audit Logging.

Worked scenario: one alert, two capabilities

A monitoring rule fires at 02:10: the export API's 200-response volume for one partner client is three times its weekly baseline. That is all monitoring can say. With observability in place, the analyst pivots on client_id in traces and finds that every request carries the same token jti, originates from a cloud network the partner never uses, and calls a bulk endpoint the partner has not called in 90 days. Pivoting on the token hash shows the same token used two days earlier at low volume, below every threshold. The trace attributes identify the issuing grant, so the team revokes the refresh token, disables the client and has a scoped list of exported records for the notification decision.

Without trace-level identity, the team would know only that volume rose. Without the volume monitor, the observable data would have sat unqueried. The Wyndham hardcoded app password breach shows the other side: when a service credential that can read customer login tokens leaks, the first question is whether it was used, and only logged, attributable access can answer it.

Decision guidance

  • Start with monitoring when: you have no on-call alerts for identity provider health, log source silence, detection pipeline freshness or authentication failure spikes. These are cheap, well understood and catch the loud failures.
  • Invest in observability when: investigations regularly stall at "we cannot tell which identity did that", your estate is microservices or APIs where one request crosses many systems, or you run agents.
  • Small organisations: use the observability features of your cloud provider or existing APM tool, add identity attributes at the gateway, and keep audit logs in the SIEM. Do not build a separate trace platform for security.
  • Large organisations: run one telemetry pipeline (for example OpenTelemetry Collectors) feeding both observability and security backends, with security-owned tail-sampling and redaction policies, rather than parallel agents on every host.
  • Regulated environments: confirm that sampled or short-retention data is never the only copy of anything a regulator or auditor may ask for.

How to implement

  1. Inventory questions, not tools: list the ten questions your last three investigations could not answer quickly. Each one becomes either a monitoring rule (known condition) or an instrumentation requirement (attribute to capture).
  2. Set the monitoring baseline: golden-signal alerts for identity, secrets and detection services; "source went silent" rules for every critical log source; freshness SLIs for the SIEM. Follow the SRE rule that every page must be actionable.
  3. Define the identity attribute standard: subject, client ID, workload identity, token identifier hash, tenant and decision outcome, emitted at the gateway and at authorisation points. Test that no raw token, cookie or password appears.
  4. Pilot on one high-value path: instrument login, token issuance and one sensitive API end to end. Run a tabletop that answers the scenario above using only the new telemetry.
  5. Write security sampling policy: tail-sampling rules that keep errors, authentication, privileged and agent tool-call traces at 100%; route the rest to cheaper storage where budget allows.
  6. Turn findings into rules: every investigation that used observability should end with a question: can this become a monitored condition? This is how observability feeds monitoring. See the Detection Engineering Guide.
  7. Roll back safely: if instrumentation causes latency or captures sensitive content, disable the specific attributes or sampling rule through collector configuration rather than removing SDKs, and keep a version history of pipeline configuration.

Detection and measurement

  • Time to answer: median minutes to answer "what else did this identity do in the last 7 days" during exercises. Target a steady fall.
  • Identity coverage of traces: share of traces on sensitive routes that carry subject and client ID attributes. Target above 95%.
  • Silent source detection time: minutes between a critical log source stopping and an alert. Target under 30.
  • Actionable page rate: share of security pages that led to an action. Low rates mean vague rules; see the Alert Fatigue Reduction Guide.
  • Observability to monitoring conversion: number of new monitored rules created from investigations per quarter.
  • Sensitive data leakage: count of seeded test secrets found in the observability backend. Target zero.

Framework mapping

FrameworkReferenceMonitoringObservability
NIST CSF 2.0DE.CM-01, DE.CM-09Networks, hardware, software and runtime environments monitored for adverse eventsTelemetry depth that makes monitoring meaningful
NIST CSF 2.0DE.AE-02, DE.AE-03Alert triggers analysisAnalysis and correlation from multiple sources
NIST SP 800-53 Rev 5SI-4, CA-7System Monitoring; Continuous MonitoringSupports both
NIST SP 800-53 Rev 5AU-6, AU-9, AU-11, AU-12Audit record review and reportingGeneration, protection and retention of the records investigations depend on
CIS Controls v8.18.2, 8.11, 13.1Collect audit logs, review them, centralise security event alertingDetailed logs and correlation feed reviews
ISO/IEC 27001:2022Annex A 8.15, 8.168.16 Monitoring activities8.15 Logging
MITRE ATT&CK v20T1685 Disable or Modify ToolsAlert on telemetry lossNot applicable once data is lost

NIST SP 800-137 frames continuous monitoring as visibility into assets, threats and the effectiveness of controls, which is the monitoring half. The draft NIST SP 800-92 Rev. 1 (initial public draft, October 2023) treats log management as generating, transmitting, storing, accessing and disposing of log data, which is the foundation both halves rest on.

Common failure patterns

  • Calling dashboards observability: a wall of predefined charts is monitoring. If you cannot ask a new question across signals by an arbitrary attribute, you do not have observability.
  • Sampling security evidence away: a uniform 1% head-sampling rate set by the platform team for cost, never reviewed by security, discovered during the first investigation that needed traces.
  • No identity on telemetry: traces show service-to-service calls but not which user, client or workload identity drove them, so the trace stops being evidence.
  • Secrets in attributes: HTTP header capture or opt-in prompt capture switched on in production, putting bearer tokens and customer data into a platform with broad read access. See the observability platform compromise playbook.
  • Not monitoring the monitors: no alert when a log source, collector or agent stops sending, so an attacker who disables tooling gets silence instead of a page.
  • Treating traces as audit: relying on observability data for compliance evidence when it is sampled, mutable and short-lived.

Practitioner checklist

  • List the questions your last investigations could not answer, and assign each to monitoring or instrumentation.
  • Alert on silence for every critical log source, collector and agent.
  • Monitor freshness and completeness of the SIEM and detection pipeline as SLIs.
  • Add subject, client ID, workload identity and token identifier hash to spans at gateways and authorisation points.
  • Restart trace context from the internet at the edge and rate-limit honouring of the sampled flag.
  • Write tail-sampling rules that keep errors, authentication, privileged and agent tool-call traces at 100%.
  • Keep audit logs on a separate, unsampled, tamper-evident path.
  • Keep GenAI content capture off by default; enable it per use case with redaction and access control.
  • Seed test secrets and confirm none reach the observability backend.
  • Review each investigation for a rule that can move from observability into monitoring.

Standards and references

Related NHI Mgmt Group resources: Observability, telemetry and SRE for security teams · SIEM Guide · LLM Observability Guide · Agent Observability vs Audit Logging · Logging and Monitoring Policy Template

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 14 November 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.