LLM Observability Guide: Tracing, Logging and Monitoring AI Applications

LLM applications fail in ways ordinary application monitoring cannot see: a request returns HTTP 200 while the answer is wrong, the prompt leaked a customer's data, a retrieval step pulled the wrong documents or a loop quietly spent a week's budget in an afternoon. LLM observability is the practice of tracing, logging and measuring every model call, retrieval and tool step so that engineers, security teams and finance can explain what happened and why. This guide covers the telemetry model (OpenTelemetry GenAI semantic conventions), how to instrument applications and gateways, how to capture prompts and responses without creating a new data breach risk, sampling and retention, quality evaluation, and the security detections that LLM telemetry makes possible. It is written for platform, SRE and security engineers running LLM applications in production. Incident response and kill switches for autonomous agents are covered in the AI Agent Observability and Incident Response Guide and are not repeated here.

Key takeaways

• Standardise on OpenTelemetry GenAI conventions. Attributes such as gen_ai.provider.name, gen_ai.request.model and gen_ai.usage.input_tokens let traces from SDKs, frameworks and gateways land in one schema and one backend.
• Prompt and response content is opt-in for a reason. The conventions warn that message attributes are likely to contain sensitive information including personal data; capture them deliberately, redact before export and store them under a shorter retention than metadata.
• Trace the whole request, not just the model call. Retrieval, tool execution and MCP calls are where many quality and security failures start, so they need their own spans in the same trace.
• Tokens are the unit of cost, latency and abuse. Token metrics per caller and model are the earliest signal of runaway loops, stolen keys and Denial of Wallet.
• The telemetry store is sensitive. Observability backends hold prompts, system instructions and API key identifiers, so they need the same access control, audit and supplier review as production data.

Why LLM observability is different

Outputs are non-deterministic, so one bad answer cannot be reproduced by replaying the request. The trace, including the exact prompt version, model version and retrieved context, is often the only evidence of what the model was given.

The telemetry model: OpenTelemetry GenAI semantic conventions

OpenTelemetry's semantic conventions for generative AI define spans, metrics, events and exceptions for model inference, agents, tools and MCP. They now live in a dedicated repository (open-telemetry/semantic-conventions-genai) and are still marked Development, so attribute names can change between releases; pin the convention version your instrumentation emits and check changelogs before upgrading.

Spans

• Inference spans: named {gen_ai.operation.name} {gen_ai.request.model}, for example chat gpt-4o. Operation names include chat, text_completion, embeddings and retrieval.
• Agent and tool spans: create_agent, invoke_agent {gen_ai.agent.name}, invoke_workflow, plan and execute_tool.
• MCP spans: named {mcp.method.name} {target}, for example tools/call get-weather, with mcp.session.id and mcp.protocol.version.
• Sampling-relevant attributes: the conventions say gen_ai.operation.name and gen_ai.provider.name should be set at span creation so samplers can use them.

Key attributes

Metrics

• Client: gen_ai.client.operation.duration (histogram with explicit buckets from 0.01 s to 81.92 s) and the inference token metrics gen_ai.client.inference.usage.input_tokens and gen_ai.client.inference.usage.output_tokens, with cache and reasoning variants.
• Model server: gen_ai.server.request.duration, gen_ai.server.time_to_first_token and gen_ai.server.time_per_output_token for self-hosted inference.
• Agents and tools: gen_ai.invoke_agent.duration, gen_ai.invoke_agent.tool_calls and gen_ai.execute_tool.duration.

Other conventions exist, notably OpenInference (used by Arize Phoenix and by Envoy AI Gateway tracing). If you run both, map them to one schema in the Collector rather than querying two vocabularies.

Where to instrument

Most mature estates use two layers: gateway telemetry for coverage and attribution, SDK telemetry for depth, joined by W3C Trace Context propagation so the gateway span becomes a child of the application trace. Kong and Envoy AI Gateway both export GenAI telemetry in OpenTelemetry form; LiteLLM exports through callbacks including otel and Langfuse. See the AI Gateway and LLM Gateway Guide.

Capturing content safely

Prompt and response content is the most useful and most dangerous telemetry you will collect. It routinely contains personal data, customer records pulled in by retrieval, secrets pasted by users and the full system prompt. OWASP LLM02:2025 Sensitive Information Disclosure and LLM07:2025 System Prompt Leakage both apply to your own logs, not only to model output.

• Default to metadata only: leave the opt-in content attributes off in production unless a use case needs them. In the OpenTelemetry Python GenAI instrumentation, message content capture is controlled by OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT.
• Know your defaults: some integrations capture content unless told not to. Envoy AI Gateway's OpenInference tracing records full requests and responses unless OPENINFERENCE_HIDE_INPUTS, OPENINFERENCE_HIDE_OUTPUTS and OPENINFERENCE_HIDE_EMBEDDINGS_TEXT are set to true. LiteLLM provides turn_off_message_logging (metadata and spend still tracked) and redact_user_api_key_info to strip hashed key, user and team identifiers.
• Redact in the pipeline, not only in one SDK: Langfuse documents that its masking functions only affect data sent to Langfuse, and other exporters receive unmodified spans. For mixed estates, redact in the OpenTelemetry Collector with the redaction processor, using allowed_keys (which fails closed: attributes not listed are removed), blocked_values regular expressions for card numbers, email addresses and key formats, and hash_function where you need to correlate without storing the value.
• Separate content from metadata: the conventions allow in-process hooks that upload content to external storage and leave only a reference on the span. Use this to put content in a store with stricter access and shorter retention than the trace backend.
• Truncate and bound: content can exceed backend attribute limits; the conventions permit truncation that preserves JSON structure. Set limits deliberately so truncation does not drop the part you need for evidence.
• Respect data subject rights: if prompts contain personal data, the trace store is in scope for retention schedules, access requests and erasure. See the AI and Privacy Guide.

Sampling, retention and cost

• Metrics are always complete: token and latency metrics should be recorded for every call, independent of trace sampling, because they drive budgets and alerting.
• Tail sampling for traces: the Collector's tail_sampling processor decides after a trace completes (default decision_wait of 30 s). Keep 100% of traces with status_code ERROR, guardrail blocks (a string_attribute or ottl_condition policy), high token counts (numeric_attribute) and slow requests (latency), and a probabilistic sample of the rest.
• Content hooks ignore sampling: the conventions say content upload hooks should run regardless of the sampling decision, which is useful for evaluation datasets but means content volume does not shrink with trace sampling.
• Tiered retention: for example, metrics for 13 months for trend and budget, metadata traces for 90 days, content for 30 days or less unless held for an investigation. Align with NIST SP 800-53 AU-11 Audit Record Retention and your privacy notices.
• Cardinality: never put user IDs, conversation IDs or prompt text into metric labels; keep them on spans. Envoy AI Gateway, for example, deliberately does not add session identifiers to metrics by default.

Quality monitoring and evaluation

• Online evaluation: score a sample of production traces for groundedness, relevance and policy compliance using LLM-based evaluators, code checks or human labels. Tools such as Arize Phoenix and Langfuse attach scores to traces.
• Datasets and experiments: turn interesting traces (failures, user complaints, red team findings) into datasets and rerun them against new prompt versions or models before release. See the AI Red Teaming Guide.
• Version everything: record prompt name and version (gen_ai.prompt.name, gen_ai.prompt.version), model version and retrieval index version on each span, so a quality drop can be tied to a change.
• Retrieval quality: record gen_ai.data_source.id and, where approved, retrieved document identifiers and scores (gen_ai.retrieval.documents is opt-in). Wrong-document retrieval is a common root cause of wrong answers and of data exposure across permission boundaries. See the RAG Security Guide.
• Evaluators are models too: an LLM judge sees the content it scores, so route it through the same approved providers and redaction as production.

Security detections from LLM telemetry

Forward these as alerts or summarised events to the SIEM rather than shipping full prompt content there. See the SIEM Guide, the Prompt Injection Guide and the LLMjacking Guide.

Where identity fits

• Attribute every span to an identity: the calling workload or agent identity and, for user-facing apps, a pseudonymous end-user identifier. Without it, telemetry tells you something went wrong but not whose credential to revoke.
• Exporter credentials are NHIs: OTLP endpoints and SaaS observability backends authenticate with API keys or tokens. Scope them to ingest only, store them in a secrets manager and rotate them; a leaked ingest key can be used to inject false telemetry, and a leaked read key exposes every captured prompt. See the Secrets Management Guide.
• Restrict read access: limit who can view content-bearing traces, log every access to them (AU-9 Protection of Audit Information) and review access quarterly.

Implementation roadmap

1. Decide the schema: adopt OpenTelemetry GenAI conventions at a pinned version and agree resource attributes (service.name, environment, owning team).
2. Deploy a Collector tier: route all GenAI telemetry through OpenTelemetry Collectors you control, with redaction and tail sampling configured before any backend.
3. Instrument the gateway first: it gives coverage, token and cost metrics for every team in one step.
4. Add SDK instrumentation to priority applications: customer-facing and data-heavy ones first, with content capture off.
5. Pilot content capture: enable it for one application in a non-production environment, review what is captured with privacy and security, tune redaction, then decide per application.
6. Build detections and dashboards: spend per caller, error and refusal rates, time to first token, guardrail verdicts, model drift from approved routes.
7. Add evaluation: online scoring on a sample, and datasets from failures feeding pre-release tests.

Testing and rollback: validate redaction rules by sending synthetic prompts containing test card numbers, email addresses and fake keys, and confirming they never reach the backend. Keep Collector configuration in version control; if a change drops required attributes or leaks content, revert the Collector configuration rather than redeploying applications.

Measuring the programme

Framework mapping

NIST AI 600-1, the Generative AI Profile of the AI Risk Management Framework, is the companion reference for governance of generative AI risks that this telemetry helps evidence.

Common failure patterns

• Full prompt capture switched on by a default. A tracing integration records every prompt and response, and the observability SaaS becomes the largest unreviewed store of customer data in the company.
• Redaction in one SDK only. A second exporter or a framework's own instrumentation sends the unmasked spans elsewhere.
• Traces without identity. Spend spikes are visible, but nobody can tell which key, agent or user caused them.
• Request-level sampling that drops the evidence. Head sampling at 1% discards the one trace showing an injection, because the decision was made before the outcome was known.
• Unpinned conventions. An instrumentation upgrade renames attributes and dashboards and alerts silently stop matching.
• Monitoring only the model call. Retrieval and tool spans are missing, so the trace shows a correct model call fed with the wrong documents.

Practitioner checklist

• Adopt OpenTelemetry GenAI semantic conventions at a pinned version across SDKs and gateways.
• Route all GenAI telemetry through Collectors you control, with redaction before any backend.
• Record token, latency and error metrics for every call, regardless of trace sampling.
• Keep content capture off by default and approve it per application with privacy and security.
• Check and set content defaults on every integration, including gateway tracing.
• Use tail sampling that keeps errors, guardrail blocks, slow and high-token traces.
• Attribute every span to a workload or agent identity, and pseudonymous users where relevant.
• Scope, vault and rotate exporter and backend API keys; audit access to content-bearing traces.
• Set tiered retention for metrics, metadata and content.
• Turn production failures into evaluation datasets and run them before each prompt or model change.

Standards and references

• OpenTelemetry: Semantic Conventions for Generative AI
• OpenTelemetry: OpenTelemetry for Generative AI
• OpenTelemetry Collector Contrib: Redaction Processor
• OpenTelemetry Collector Contrib: Tail Sampling Processor
• OWASP: Top 10 for LLM Applications 2025
• NIST: AI 600-1 Generative Artificial Intelligence Profile
• NIST: SP 800-53 Rev. 5
• Langfuse: Masking of Sensitive LLM Data

Related NHI Mgmt Group resources: AI Agent Observability and Incident Response Guide · AI Gateway and LLM Gateway Guide · Cloud Logging and Audit Trail Guide · Generative AI Data Loss Prevention Guide · LLM Security Guide