Start with service level objectives for the business outcomes that matter most, then map them to observable signals. Track availability, latency, error rate, and throughput, and add golden signals for saturation and traffic. Pair those baselines with service specific dashboards so engineering, operations, and business teams can see the same system through different lenses.
Why This Matters for Security Teams
Monitoring baselines are not just an operations concern. In AWS and OpenTelemetry environments, they are often the earliest way to spot misconfiguration, credential misuse, data exfiltration, or a service boundary that has shifted without approval. A baseline that reflects expected traffic, log volume, span patterns, and error behavior gives security teams a practical reference point for deciding what is normal, what is degraded, and what deserves escalation.
That matters because observability stacks can create a false sense of coverage. High telemetry volume does not equal useful monitoring if the signals are noisy, incomplete, or disconnected from business services. Security teams should treat the baseline as a control surface, not just a dashboard exercise, and align it with logging retention, alert thresholds, and incident response paths. Current guidance on control design remains stronger than guidance on exact metric values, so teams need to define baselines per service, per environment, and per trust boundary. For control mapping, NIST SP 800-53 Rev 5 Security and Privacy Controls remains a useful anchor for log management, monitoring, and response expectations. In practice, many security teams discover weak baselines only after alert fatigue or a real incident has already exposed the blind spots.
How It Works in Practice
Effective baselines start with defining what “normal” looks like for each service, not for the platform as a whole. In AWS, that usually means separating control plane activity from workload telemetry, then deciding which signals are stable enough to baseline and which are too bursty to be meaningful. OpenTelemetry adds another layer: traces, metrics, and logs must be correlated so a spike in latency, a failed auth path, or an unusual retry pattern can be interpreted in context.
A practical baseline usually combines four layers:
- Business service targets, such as availability and user-facing latency.
- Platform signals, such as API error rates, queue depth, CPU saturation, and network anomalies.
- Identity and access signals, such as unusual role assumptions, token use, or privilege changes.
- Telemetry hygiene signals, such as missing spans, dropped logs, or sudden collector failures.
Teams should baseline by environment because production, staging, and disaster recovery will not generate the same patterns. They should also baseline by service tier because a customer-facing API and an internal batch job produce very different observability shapes. Where possible, use change windows to explain temporary shifts, then verify whether the shift returns to expected ranges after release or scaling events.
OpenTelemetry is most useful when instrumentation standards are consistent enough to compare services without flattening their differences. AWS native telemetry can then be used to fill gaps around infrastructure and control plane visibility. For control expectations on logging, monitoring, and incident handling, NIST Cybersecurity Framework is a practical companion to the technical stack. These controls tend to break down when teams mix ephemeral serverless workloads, autoscaling containers, and inconsistent trace sampling because the “normal” range changes too quickly to distinguish genuine anomalies from workload churn.
Common Variations and Edge Cases
Tighter baseline definitions often improve detection quality, but they also increase tuning overhead and can create false positives when services are highly dynamic. Teams need to balance precision against operational cost, especially in platforms where workload shape changes throughout the day or across customer cohorts.
Some edge cases need special handling. Bursty event-driven services may need percentile-based thresholds instead of fixed limits. Multi-tenant platforms may require per-tenant segmentation so one noisy tenant does not mask another tenant’s outage or attack signal. Compliance-heavy environments may also need separate baselines for audit logging and application telemetry because those streams serve different purposes and retention rules.
There is no universal standard for exact baseline formulas yet, so best practice is evolving. The strongest programs document how thresholds were chosen, who owns them, and what change approval is required before they move. That is especially important when observability data is also used for security analytics, because security teams can inherit operational blind spots if the baseline logic was designed only for performance management. Where identity and access are part of the question, a stable baseline should also include expected IAM and NHI activity so unusual automation or role usage stands out quickly. In practice, monitoring programs fail most often when metric ownership is split across teams and nobody is accountable for revisiting thresholds after architecture changes.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST CSF 2.0 provides the primary governance reference for this topic.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | DE.CM-1 | Continuous monitoring depends on defined, repeatable baseline signals. |
Establish monitoring baselines and review them as part of continuous security monitoring.
Related resources from NHI Mgmt Group
- What is the difference between monitoring and observability for IAM teams?
- How should teams classify AWS permissions that change monitoring or session behaviour?
- How should security teams implement OpenTelemetry for AI observability in multi-agent systems?
- How should teams implement LLM observability when they need both infrastructure monitoring and output quality control?
