Observability & Monitoring

Traditional software observability tells you whether your system is up and fast. AI observability must also tell you whether your system is good — whether the outputs are correct, safe, and useful. This module covers why AI monitoring differs from traditional monitoring, how to log and trace AI interactions, what metrics to track, which tools to use, and how to build dashboards and alerts that catch quality degradation before your users do.

Why AI Observability Is Different

In traditional software, if the API returns a 200 status code with a well-formed response, the request was successful. In AI systems, the API can return a 200 with a perfectly structured response that is completely wrong, hallucinated, or harmful. Success is no longer a binary — it's a spectrum of quality that requires new monitoring approaches.

Logging AI Interactions

Every AI interaction should be logged with enough detail to reconstruct what happened, why it happened, and how much it cost. This data powers debugging, evaluation, cost analysis, and compliance.

Structured Logging Schema

Cost Tracking

Cost tracking deserves special attention because AI API costs are per-token and can vary dramatically across requests. A single request with a large context window can cost 100x more than a simple query.

• Per-request cost: Calculate input tokens multiplied by the model's input price, plus output tokens multiplied by the output price. Log this on every request.
• Per-user cost: Aggregate request costs by user to identify heavy users, detect abuse, and inform pricing decisions.
• Per-feature cost: Tag requests by feature (search, chat, summarization) to understand which features drive cost and where optimization will have the most impact.
• Cost budgets: Set daily and monthly cost budgets with alerts at 50%, 80%, and 100% thresholds. Consider automatic rate limiting or model downgrading when approaching budget limits.

Tracing Agent Workflows

Modern AI applications — especially agents and multi-step RAG pipelines — involve chains of operations: retrieval, multiple LLM calls, tool executions, and conditional branching. Tracing provides visibility into these complex workflows by linking related operations into a tree of spans.

Spans and Traces

Borrowing from distributed tracing (OpenTelemetry), AI observability uses the same concepts:

• Trace: The complete lifecycle of a user request — from initial query to final response. A trace contains one or more spans.
• Span: A single operation within a trace — an LLM call, a vector search, a tool execution, or a guardrail check. Each span records its start time, duration, inputs, outputs, and status.
• Parent-child relationships: Spans are nested. An agent trace might contain a top-level "agent_run" span with child spans for "planning", "retrieval", "tool_call", and "generation" — each revealing where time and tokens are spent.

Key Metrics to Monitor

Latency Metrics

• Time to first token (TTFT): How long until the user sees the first token of the response. Critical for streaming UIs — users perceive responsiveness by when output starts, not when it finishes.
• Total latency: End-to-end time for the full response. Track p50, p95, and p99 percentiles — averages hide tail latency problems.
• Tokens per second: Generation speed. Important for self-hosted models where throughput is a function of GPU utilization and batching.

Cost Metrics

• Cost per request: Average and p95 cost. High variance indicates some requests are disproportionately expensive.
• Daily/weekly spend: Track trends and detect spikes early. A sudden 3x increase in daily spend might indicate a bug causing unnecessary retries or inflated prompts.
• Cost per user action: The business-level metric — how much does it cost to serve one customer support resolution, one document summary, or one search query?

Quality Metrics

• User feedback scores: Thumbs up/down ratios, CSAT, or custom rating scales. The most direct signal of output quality.
• Guardrail trigger rate: How often do safety filters or content policies activate? A sudden increase may indicate the model is producing more problematic outputs.
• Retry and regeneration rate: How often do users retry or regenerate? High retry rates signal that outputs aren't meeting user expectations.
• Online eval scores: Run automated evaluations (e.g., LLM-as-judge) on a sample of production traffic to continuously measure quality dimensions like correctness and relevance.

Observability Tools

Building Dashboards

A well-designed AI dashboard gives your team at-a-glance visibility into system health, quality, and cost. Structure your dashboard around three levels of detail.

Executive Dashboard

High-level metrics for leadership and product managers:

• Total AI requests per day/week (trend line)
• Total spend and cost per user action (trend line)
• Overall user satisfaction score (trend line)
• Error rate and escalation rate

Engineering Dashboard

Operational metrics for the engineering team:

• Latency percentiles (p50, p95, p99) by endpoint
• Token usage by model and feature
• Guardrail trigger rate and types
• Request volume heatmap by hour
• Model provider status and error rates

Quality Dashboard

Output quality metrics for AI/ML engineers:

• Online eval scores over time (by eval dimension)
• User feedback distribution (thumbs up/down, ratings)
• Retry and regeneration rates
• Top failure categories from user reports
• Embedding drift scores (for RAG systems)

Alerting on Quality Degradation

Alerts for AI systems must go beyond uptime and latency. Quality degradation often happens gradually — a few percentage points per week — making it invisible without statistical alerting.

Alert Categories

• Availability alerts: Model API errors, timeout rates, and circuit breaker activations. Threshold: error rate > 1% over 5 minutes.
• Latency alerts: p95 latency exceeds SLA. Threshold: p95 > 5 seconds for 10 minutes (adjust based on your application).
• Cost alerts: Hourly or daily spend exceeds budget. Use anomaly detection rather than fixed thresholds — a 3x spike on a Tuesday when traffic normally dips is more alarming than steady high spend on a Monday.
• Quality alerts: User feedback score drops below threshold. Online eval scores decline by more than 5% week-over-week. Guardrail trigger rate increases significantly.
• Drift alerts: For RAG systems, monitor whether the distribution of retrieved documents or embedding similarities is shifting. This can indicate stale data or degraded retrieval.

Cost Anomaly Detection

Cost anomalies deserve dedicated attention because they can be both a symptom of bugs and a direct financial risk:

• Infinite loop detection: An agent stuck in a retry loop can burn through thousands of dollars in minutes. Alert on any single trace exceeding a cost threshold (e.g., $1 per trace).
• Prompt inflation: A code change that accidentally includes too much context can multiply costs. Monitor average input tokens per request and alert on sudden increases.
• Model routing failures: If your routing logic fails and all requests go to the most expensive model, costs spike. Monitor the distribution of requests across models.

Resources