What is LLM observability?

LLM observability is production monitoring for AI systems that combines traces, token usage, latency, quality scores, prompt versions, human feedback, and business outcomes. Its job is to identify user-visible failure patterns before they become incidents.

Which LLM production metrics should teams alert on first?

Start with cost per resolved task, hallucination rate, p95 latency, tool-call success rate, retrieval recall drift, prompt-template drift, and negative human feedback. Each metric needs an owner and a response runbook.

How do you detect prompt drift in production?

Track prompt version, rendered-prompt hash, token-length distribution, refusal rate, output length, and embedding shift. Population Stability Index is a common first detector, with 0.10 as an investigation threshold and above 0.25 as major drift.

Should production LLM traces capture prompts and responses?

Only with explicit controls. OpenTelemetry GenAI payload attributes are opt-in because prompt and response capture can expose PII, so production systems should default payload capture off or redact through a gateway.

LLM Observability Must Catch Drift Before Incidents

The short answer: LLM observability should alert on user-visible failure signals first, but raw traces and token dashboards won't catch prompt drift by themselves.

The failure mode has changed. A normal API endpoint usually breaks with a 500, a timeout, or a saturation graph. A production LLM feature can keep returning 200s while cost doubles, citations get weaker, a tool loop burns tokens, or a prompt edit quietly changes the product.

TL;DR: Treat LLM observability as AI quality monitoring attached to ordinary SRE telemetry. Instrument every model call with OpenTelemetry GenAI attributes, then alert on cost per resolved task, hallucination rate, p95 latency, tool success, retrieval drift, prompt drift, and negative human feedback. The operating rule is simple: no owner, no runbook, no alert.

Key Takeaways

OpenTelemetry GenAI is the practical schema for LLM traces as of June 2026, but the GenAI conventions are still marked Development/Experimental.
Prompt drift needs version IDs, rendered-prompt hashes, token-length distributions, and output-distribution checks.
Hallucination detection works best as sampled online evaluation joined to retrieved context, not as a blanket judgment on every response.
Cost alerts should use dollars per successful task, not total tokens.
Payload capture in production is a governance decision because prompts and outputs can contain personal data.
Canary releases for prompts need online eval gates and runtime rollback, especially for agents with tool access.

Why LLM Observability Starts With Drift

LLM observability is the practice of tracing, measuring, and evaluating production AI behavior across model calls, prompts, tools, retrieval, user feedback, and business outcomes. The goal is early detection of quality, cost, latency, and safety regressions that standard service metrics miss.

That definition matters because LLM systems fail across layers.

A retrieval change can reduce factual grounding while latency stays flat. A prompt patch can increase refusals by five percentage points. An agent can retry a failing tool until a single task costs 10 times its baseline.

The strongest production setup watches the symptom the user feels, then follows the trace backward.

Failure signal	Best first metric	Trace detail you need	Default response
Answers become unsupported	Hallucination rate / faithfulness	Retrieved context span + output	Stop rollout, inspect prompt and retriever
Prompt behavior shifts	PSI on prompt distribution	Prompt version + rendered hash	Compare canary to control
Cost spikes	Cost per resolved task	Input/output tokens + outcome	Cap tenant, roll back prompt or agent
Agent gets stuck	Tool success and loop count	TOOL spans + agent spans	Trip circuit breaker
Users reject answers	Negative feedback rate	`trace_id` joined to feedback	Promote samples into eval set

What Is Current in June 2026?

OpenTelemetry is the safest backbone to choose now. The Cloud Native Computing Foundation announced OpenTelemetry’s graduation on May 21, 2026, which makes it a credible default for teams standardizing traces across normal services and LLM systems.

For GenAI specifically, the OpenTelemetry GenAI semantic conventions define the gen_ai.* namespace for model calls, token usage, operation duration, events, and trace attributes. The important caveat: the GenAI conventions are still labeled Development/Experimental as of June 2026.

That means you should adopt them, but pin versions and review release notes.

The GenAI metrics sub-doc renders against v1.42.0 in May 2026, while the agent and framework spans render against v1.41.1.gen-ai in June 2026. That agent update split invoke_agent into invoke_agent_client and invoke_agent_internal, so dashboards keyed to the old span name need migration.

Every LLM span should include the required attribute triple from the GenAI span conventions: gen_ai.operation.name, gen_ai.provider.name, and gen_ai.request.model.

Add prompt metadata yourself. The standard gen_ai.* attributes won’t magically tell you which prompt template, canary, or feature flag produced a bad answer.

yaml

llm_span_required:
  gen_ai.operation.name: chat
  gen_ai.provider.name: openai
  gen_ai.request.model: current-production-model
  gen_ai.prompt.version: v3.2.1
  gen_ai.prompt.hash: sha256(rendered_template)
  app.prompt.environment: canary
  app.tenant_id: tenant_hash
  app.outcome: resolved

The Seven LLM Production Metrics That Matter

The useful question is never “what can we graph?” It’s “what action will someone take when this moves?”

Here is the minimum production set.

Metric	Why it catches incidents early	Suggested alert
Cost per resolved task	Finds agent loops and expensive prompt regressions	>25% over trailing 7-day baseline for 1 hour
Hallucination rate	Finds unsupported answers before support tickets spike	Baseline + 2σ or +5 percentage points
p95 end-to-end latency	Tracks user-visible response delay	2x SLO burn for 30 minutes
Tool-call success rate	Finds failing dependencies inside agents	<99% for non-retry tools over 15 minutes
Retrieval recall drift	Finds RAG quality loss before answer quality collapses	>5 percentage point drop vs last green release
Prompt-template drift	Finds prompt/input distribution changes	PSI ≥0.10 investigate, ≥0.25 alert
Negative human feedback	Captures product reality judges miss	Baseline + 2σ or 1.5x trailing 7-day rate

For token and latency telemetry, use the OpenTelemetry GenAI metrics such as gen_ai.client.operation.duration and gen_ai.client.token.usage.

For quality telemetry, attach eval scores to traces. Ragas faithfulness decomposes an answer into statements and checks whether each is supported by retrieved context. That makes it a good hallucination detection primitive for RAG workloads.

Prompt drift PSI thresholds

How Do You Detect Prompt Drift?

Prompt drift has two forms.

Input drift means the prompts entering the system changed. Output drift means the model’s behavior changed for similar inputs.

Track both.

For input drift, compute Population Stability Index over prompt-token-length buckets, rendered-prompt hash distribution, or a prefix hash of the first 128 tokens. Arthur documents PSI as a model monitoring metric in its Population Stability Index guide, and Evidently documents related drift customization in its data drift docs.

Use the standard thresholds carefully:

PSI range	Production interpretation
<0.10	No meaningful prompt distribution drift
0.10-0.25	Investigate, compare segments and prompt versions
>0.25	Major drift, consider rollback or traffic hold

For output drift, watch refusal rate, average output token length, JSON parse failures, citation precision, and embedding centroid shift. Arize documents embedding drift, and Phoenix positions itself as an OpenTelemetry-native option in the Phoenix docs.

The practical pattern is warehouse-first. Emit span metadata, then compute drift over five-minute or hourly windows in BigQuery, Snowflake, ClickHouse, or your metrics store.

The Contrarian Metric: Total Tokens Are Usually Noise

Total tokens used feels important because it maps to a bill.

It’s a weak alert. Traffic growth, a marketing campaign, or a product launch can all move total tokens without indicating a failure.

The stronger metric is cost per resolved task. It normalizes spend by outcome, which is what finance, product, and engineering all care about.

A support assistant that spends 30% more tokens while resolving 50% more tickets may be a good release. An agent that spends 2x more tokens with flat success rate is probably looping, over-retrieving, or over-answering.

This is also where AI monitoring needs business events. Without “resolved,” “converted,” “accepted,” “escalated,” or “edited,” your LLM production metrics can’t distinguish useful work from expensive motion.

What Should LLM Traces Capture?

An LLM trace should show the whole path from user request to final outcome: app span, retriever spans, model spans, tool spans, agent reasoning spans, eval scores, and feedback events.

The OTel GenAI spec includes payload attributes such as gen_ai.input.messages and gen_ai.output.messages, but they are opt-in. The GenAI events conventions call out the payload risk because production prompts and responses can contain sensitive data.

Default production posture: capture metadata, redact payloads, and sample full content only for approved debugging and eval workflows.

LangSmith documents OpenTelemetry tracing in its OTel tracing guide and separately documents OTel gateway trace redaction. Datadog’s LLM observability OTel instrumentation follows the same broad pattern: instrument richly, control sensitive payloads at the boundary.

For agents, make tool calls first-class spans. A single top-level “chat” span hides the expensive part of the system.

How Should You Ship Prompt Changes Safely?

Prompt changes deserve release engineering.

Use semantic versioning, immutable content hashes, and environment promotion. Braintrust documents this pattern in its prompt versioning and deployment guide, and LangSmith covers observability primitives in its LangSmith observability docs.

A safe rollout looks like this:

Promote the prompt to canary.
Route 1-5% of production traffic using deterministic hashing on user_id or session_id.
Score canary traffic with the same online eval pipeline as production.
Require at least 200 samples or 30 minutes, whichever is longer.
Roll back automatically on cost, latency, hallucination, or negative-feedback burn.

For agents with side effects, run shadow mode before canary. Send the same input through the new prompt, log the result, score it offline, and keep it away from users until the tool behavior is understood.

Feature flag metadata matters here. The OpenTelemetry feature flag event conventions give you a place to record which prompt variant a user saw, which is essential during incident review.

Governance Is Part of Observability Now

If you operate in regulated domains, logs are no longer just an engineering convenience.

The EU AI Act’s Article 12 on record-keeping and Article 19 on automatically generated logs create logging obligations for high-risk AI systems. The European Commission’s AI Act overview states penalties can reach €35 million or 7% of worldwide annual turnover for the most serious infringements.

That raises the bar for LLM traces.

You need enough telemetry to reconstruct behavior, but you also need redaction, retention policy, access control, and PII scanning. The observability stack becomes a data processor if it stores user prompts and model outputs.

A practical compromise: store structured metadata by default, store payloads only behind sampling and redaction, and preserve trace IDs that let approved reviewers retrieve the narrow evidence needed for evals or audits.

Vendor Choice Is Less Important Than Data Shape

LangSmith, Braintrust, Arize Phoenix/AX, WhyLabs, and Future AGI all cover much of the modern LLM observability surface as of June 2026.

The decision should hinge on integration fit.

Need	Strong fit
LangChain / LangGraph tracing, gateway controls, spend caps	LangSmith
Eval-as-code, prompt experiments, human review	Braintrust
Open-source OTel-native tracing and eval	Arize Phoenix
Guardrail-style checks for PII, toxicity, hallucination	WhyLabs LangKit
Multi-agent observability patterns	Future AGI

Braintrust publishes active product notes in its changelog. WhyLabs documents its AI Control Center in the WhyLabs docs. Future AGI keeps a running release notes page.

Pick the vendor that preserves trace portability. Your core schema should survive a platform migration.

Implementation Checklist

Use this before promoting any LLM feature to production.

Wire OTel SDKs and framework instrumentation for your model client.
Verify gen_ai.operation.name, gen_ai.provider.name, and gen_ai.request.model on every span.
Add gen_ai.prompt.version, prompt hash, environment, tenant, and outcome attributes.
Export gen_ai.client.operation.duration and token usage metrics.
Keep production payload capture off unless redaction and access controls are approved.
Sample 1-5% of traces for online evals, stratified by model and prompt version.
Attach human feedback to trace_id.
Track cost per resolved task, not total tokens.
Add canary gates for hallucination, latency, cost, tool success, and negative feedback.
Document an owner and runbook for every alert.

What This Means for You

If you’re starting from zero, don’t build a giant LLM observability program first.

Instrument the trace shape, add prompt versioning, and ship seven alerts with owners. Then add eval depth where risk justifies it: RAG faithfulness for knowledge workflows, tool-span success for agents, and human feedback loops for product-facing assistants.

The durable lesson is that LLM observability is production control for probabilistic software. Catch prompt drift, hallucination risk, cost spikes, and tool failures while they are still small enough to roll back.

LLM Observability Metrics That Catch Drift Early