AI Operations

LLM Observability: Traces, Quality, Cost, and Production Decisions

A production observability model that connects LLM traces, retrieval and tool behavior, quality signals, cost, alerts, and incident response.

April 8, 20264 min readLLM ObservabilityObservability

Logging prompts and responses creates a history, but it does not create operational control. When an AI workflow fails, the team needs to know which version ran, what evidence was retrieved, which tools were called, whether policy checks passed, how much the request cost, and where time was spent. LLM observability connects that execution record to quality and business behavior. It should help the team answer three practical questions: What happened? Did it produce an acceptable outcome? What decision should we make now?

Model every request as a connected trace

I assign one trace identifier to the complete user journey and represent important operations as spans:

request intake, user context, and selected workflow;
query transformation and source scope;
retrieval candidates, filters, and reranking;
model calls with configuration, token usage, and latency;
tool calls with validated inputs and structured outcomes;
safety, privacy, and authorization checks;
final answer, citations, fallback, and user feedback.

This hierarchy matters because a slow final response may come from retrieval, an external tool, retries, or generation. A bad answer may originate in a stale source rather than the prompt. A trace should preserve enough lineage to compare a healthy and failing request without requiring access to raw sensitive content. Version metadata is essential. Prompt, model, retriever, tool schema, evaluation policy, and application release should be identifiable. Otherwise, an incident cannot be linked confidently to a change.

Combine operational, quality, and outcome signals

Infrastructure metrics remain necessary: request volume, error rate, saturation, and latency percentiles. LLM workflows add other dimensions:

Dimension	Useful signals	Diagnostic value
Execution	Tool retries, routing, fallback, completion state	Reveals broken orchestration
Evidence	Retrieval coverage, citation validity, source freshness	Separates data and retrieval failures
Quality	Groundedness, task completion, expert correction	Shows whether output is acceptable
Efficiency	Tokens, model calls, cache use, cost per task	Connects spend to successful work
Safety	Policy blocks, redaction, escalation, authorization	Exposes control behavior
Product	Re-asks, abandonment, accepted outcomes	Links technical quality to usage

Metrics should be segmented by workflow, intent, language, source family, model version, and risk tier. Global averages can improve while one critical workflow regresses. The threshold for action should come from an established baseline and product risk, not from a universal number.

A useful dashboard connects system behavior to a release, a workflow segment, and an operational decision.

Preserve diagnostic value without collecting everything

Full prompts, retrieved documents, and model responses may contain personal, confidential, or regulated data. Observability design therefore needs data minimization:

classify fields before instrumentation;
redact or hash sensitive attributes where possible;
separate operational metadata from protected payloads;
apply access controls and retention by data class;
sample content only when it has a defined diagnostic purpose;
preserve user and source permissions in trace access.

More logging is not automatically better. High-cardinality payloads increase cost and make analysis harder. I prefer structured events, bounded attributes, and links to protected evidence when deeper investigation is authorized. Evaluation data needs similar care. Model-judge scores should include evaluator and rubric versions. Human feedback should retain its context. A quality score without provenance can drift when the judge, rubric, or sample changes.

Build alerts around decisions and failure clusters

An alert should identify an action, not only a metric movement. Useful alert payloads include the affected workflow and segment, candidate and baseline versions, sample size, representative traces, likely failure layer, owner, and safe response. Different failures require different policies:

deterministic security or authorization violations may require immediate blocking;
integration failures may activate a read-only fallback;
release regressions can stop a rollout;
gradual quality drift may trigger investigation and dataset refresh;
cost growth without outcome improvement may trigger routing or prompt review.

The evaluation regression blueprint covers baseline and alert design in more depth. Observability supplies the production signals and lineage that make those comparisons credible.

Run incident response from evidence

During an incident, I first define the affected journey and time window. Then I compare healthy and failing traces, separate source, retrieval, orchestration, tool, generation, and policy failures, and identify the last known good versions. A correction is replayed on representative traces and the regression suite before traffic expands again. This process should produce a durable artifact: root cause, impact, remediation, new test, owner, and monitoring change. If the same failure can recur without a new detector or regression case, the incident loop is incomplete. The public DAISI case study documents tracing, evaluation, integration tests, load testing, and security as distinct operating concerns. It illustrates the need for connected evidence without exposing private production traces.

Tradeoffs and a pragmatic rollout

Observability introduces overhead. Rich traces increase storage and privacy risk. Synchronous evaluation adds latency. Excessive sampling can inflate cost, while sparse sampling can miss rare failures. Model judges are scalable but imperfect; human review is valuable but limited. The design must balance diagnostic value against those constraints. I start with trace identifiers, version metadata, latency, errors, token usage, tool outcomes, and safe fallbacks. Next, I add sampled quality evaluation and segment-level dashboards. Alerts begin in shadow mode so noise and evaluator instability can be corrected before they page or block releases. Finally, incident reviews feed new regression tests and clearer runbooks. The objective is not a dashboard with the most charts. It is an operating system that lets the team connect a user outcome to its execution, diagnose the responsible layer, and make a safe release or recovery decision.

Sources and references

OpenTelemetry semantic conventions for generative AIStandardized attributes and events for generative AI systems
LangSmith observability documentationTracing concepts for LLM applications, retrieval, tools, and feedback

From principles to shipped systems

These articles document the methods behind my work. The project case studies show how I apply them across enterprise agents, RAG, and MLOps.

Explore project case studies View my CV

Continue exploring

Related field notes on the architecture, evaluation, and operating decisions behind production AI systems.

April 13, 20264 min readAI EvaluationRegression Testing