AI Agent Observability: What to Trace, Log and Alert On

When an agent does something wrong in production, the first question is what it actually did. Without the right telemetry, answering it means reading raw logs line by line, and sometimes it cannot be answered at all.

An agent run is not a request. It is a tree: a plan, several model calls, tool calls that reach into other systems, retries, perhaps a handoff to another agent, and a final answer. Logs written line by line from each component flatten that tree, and the one thing you need, the chain from a user's request to the record your agent changed in the ERP, is exactly what gets lost.

Observability for agents means being able to rebuild that chain for any run, months later, without storing everything forever. This post sets out what to record, using the OpenTelemetry conventions for generative AI so the data is not locked to one vendor; what to keep out of the default logs; which signals to alert on; and how long to keep it.

The unit is the trace, not the log line

OpenTelemetry models work as traces made of spans. Each span is one operation with a start, an end, a status and a set of attributes, and each knows its parent. A trace is every span that served one piece of work, across every service it touched.

Context crosses service boundaries in the W3C Trace Context traceparent header, so a span created in one service can name its parent in another. Since its July revision the MCP specification documents the same convention for MCP requests: trace context rides in each request's _meta field. A tool call handled by an MCP server can therefore join the agent's trace instead of starting a new one.

For an agent, that means one trace per run. The run is the root span. Every model call, tool call and handoff is a child. When something goes wrong you open one trace and read the run in order, instead of joining five log files on timestamps.

The shape of an agent trace

OpenTelemetry's semantic conventions for generative AI give model calls, tool calls and agents standard span names and attribute names. The project's case for them is lock-in: telemetry from agents should not be trapped in "vendor or framework specific formats." They are still marked Development, and in June they moved out of the core conventions into a repository of their own, so expect renames and pin the version your instrumentation emits.

The shape is settled enough to build on. An agent run is an invoke_agent span. Each model call is a span named for its operation and model, such as chat followed by the model name. Each tool call is an execute_tool span named for the tool. Conventions for MCP itself arrived in January, so a call to an MCP server shows up as a span such as tools/call followed by the tool name.

A typical run looks like this:

invoke_agent invoice-triage
├── chat <model>
├── execute_tool get_invoice
├── execute_tool match_purchase_order        status: error
├── chat <model>
└── execute_tool create_review_task

Read top to bottom, that trace already answers the questions an operations lead asks after an incident. What did it look at? Which call failed? What did it do about the failure? Here the matching tool failed, the model was asked again, and the agent routed the invoice to a person instead of guessing. That is the behaviour you wanted, and now you can prove it happened.

Instrumenting a tool call takes a few lines. The attribute names are the ones the conventions define, so any backend that understands them can group and chart them:

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("invoice-agent")

def run_tool(tool, call_id, arguments):
    with tracer.start_as_current_span(f"execute_tool {tool.name}") as span:
        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", tool.name)
        span.set_attribute("gen_ai.tool.call.id", call_id)
        try:
            return tool.run(**arguments)
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR))
            raise

What to record at each level

The conventions define most of what you need. At each level, record:

Two kinds of field are worth adding under your own namespace, because no convention covers them and every incident review needs them:

• •Versions. The agent's release, the prompt or instruction version and the tool set it was given. Without these you cannot tell whether a change in behaviour came from your deploy or from the model.
• •Human steps. Who approved, rejected or edited the agent's output, and when. That is the accountability record, and it is also your best quality signal.

Keep content out of the default path

The expensive mistake is logging every prompt and every response by default.

The GenAI conventions make prompts, responses and system instructions opt-in attributes, flagged as likely to contain sensitive and personal data. Instrumentations "SHOULD NOT capture them by default, but SHOULD provide an option for users to opt in." Tool call arguments and results are opt-in for the same reason. For production, the conventions recommend storing content outside the trace and recording a reference on the span, because separate storage "enables separate access controls."

A workable split:

• •Always record metadata: span names, models, token counts, tool names, statuses, durations, versions. It is small, cheap and rarely sensitive.
• •Record content selectively: sample it, redact known personal data before it leaves the process, and store it apart from the metadata, with shorter retention and tighter access. Link the two by trace ID.
• •Never record secrets: tool arguments sometimes carry tokens or credentials. Scrub them at the instrumentation layer, not in the backend.

Signals worth alerting on

The conventions also define metrics, so not everything has to be derived from traces: operation duration as a histogram in seconds, time to first token on the serving side, and token usage. The token metrics are a reminder that these names still move. In September the GenAI repository replaced the original token histogram with separate input and output instruments, in a change not yet released.

Traces give you the raw material for a small set of alerts, worth having before a customer finds the problem for you:

1. Cost per run and per day, against a budget you set. A runaway loop shows up here first.
2. Steps per run. A run that makes far more model calls than usual may be stuck in a loop.
3. Tool error rate, per tool. Tool calls fail for ordinary reasons: an expired credential, a changed API, a timeout.
4. Finish reasons. A rise in responses cut off at the token limit, or stopped by a content filter, means the inputs or the model changed.
5. Response model. If the model that answered is not the model you pinned, you want to know the same day. This is the cheapest defence against the silent behaviour changes vendors ship under an unchanged model name.
6. Human override rate. When reviewers start correcting more outputs, quality has moved, whatever the dashboards say.
7. Latency at the slow end, per agent, not just the average.

Set thresholds from your own first weeks of data rather than from anyone's benchmark. What matters is change against your own baseline.

Close the loop with evaluation

Observability tells you what happened. Evaluation tells you whether it was right. The two should feed each other.

• •Sample production traces into your evaluation set every week, weighted towards runs that failed, ran long or were overridden.
• •When you change the model, the prompt or a tool, replay the evaluation set before you ship.
• •Turn every incident trace into a regression case.

If the agent began life as a proof of concept with a written acceptance test, that test is the seed of this set. We made the case for writing it first in what an AI proof of concept must prove. The evaluation layer in our LLM architecture guide covers how to structure the harness itself.

How long to keep it

Keep metadata long enough to answer the questions people actually ask: a customer complaint, an auditor's sample, a quarter-on-quarter cost review. Keep content for less time, and only as much as evaluation and incident review need.

If any of your agents could fall into the EU AI Act's high-risk categories, the law sets a floor. High-risk systems must "technically allow for the automatic recording of events (logs) over the lifetime of the system," and providers and deployers must each keep those logs, to the extent they control them, for at least six months unless other law, data protection law in particular, provides otherwise. The amending regulation published in July moved those obligations back. They now apply from 2 December 2027 for systems that are high-risk under Annex III, and from 2 August 2028 for those under Annex I.

If none of your agents falls into those categories, that floor does not apply. The trace is still cheaper to build now than to retrofit when a customer or a regulator asks for it.

Where to start

1. Wrap each run, model call and tool call in a span using the GenAI conventions.
2. Propagate trace context to every service and MCP server the agent calls, and log every tool call with its caller and outcome, a control we also listed in our MCP server security checklist.
3. Build two views: cost and errors per agent, and a trace search by conversation or user.
4. Add the alerts above, with thresholds from your own baseline.
5. Start the weekly loop from traces into evaluation.

None of this needs a new platform. It needs a decision, made early, that every agent ships with its trace. Our AI development work scopes evaluation and monitoring before the build starts, and wires the alerts into the observability stack you already run.

Running an agent in production that you cannot fully explain yet? Contact us and we will help you instrument it before the next incident does it for you.