Coding agents like Claude Code, Codex or Cursor where costs suddenly explode, or the agent simply stops completing tasks halfway through, no visibility into what happened, what it cost, or why it failed.

None

This is a solved problem in traditional software, Twitter's engineering team described their observability stack in 2013, one of the first times the term was applied to IT systems.

Observability at Twitter

Observability at Twitter

x.com

Peter Bourgon formalized the taxonomy of metrics, tracing, and logging in 2017

None

And that model became the foundation of every observability platform you use today, everything I'll explain in this article comes from that lineage.

OpenTelemetry GenAI Semantic Conventions

OpenTelemetry is the open standard for observability. Traces, metrics, logs, one format that works across Datadog, Grafana, Honeycomb, whatever backend you use. If you've worked with any of those tools, you've already consumed OTel data.

GenAI Semantic Conventions extends that standard to generative AI. A shared vocabulary, exact attribute names, span types, metric names, agreed upon across the ecosystem.

Repo: https://github.com/open-telemetry/semantic-conventions-genai

Instead of LangChain recording llm_token_count, Claude Code / Codex recording token_usage, and your wrapper recording tokens, everyone uses:

None

That's the core of it, a dictionary the whole ecosystem commits to, over 33 libraries already have reference implementations:

None

Three signal types

Spans

A span is an operation with a start, an end, and a duration. It's the basic unit of a trace.

The key idea: a single request to your agent isn't one operation, it's several nested ones. The agent starts, calls the LLM, executes a tool, calls the LLM again. Each of those is a span, and all of them hang from the parent span that contains them.

The spec gives every operation type a standard name. The three you'll use every day:

  • gen_ai.invoke_agent: the full execution of an agent
  • gen_ai.inference.client: a call to an LLM
  • gen_ai.execute_tool: a tool call made by an agent
None

There are more for embeddings, retrieval (RAG), multi-agent workflows, planning, and memory, all following the same naming pattern:

{operation} {model or agent name}

So chat claude-fable-5 reads in any trace viewer without documentation.

Metrics

Spans tell you what happened in one request. Metrics tell you what's happening across all of them.

Every instrumented operation feeds its numbers into aggregates: how many tokens, how long it took. That's what powers your cost dashboard, your latency alerts, your capacity planning. The two you'll use every day:

  • gen_ai.client.token.usage: tokens consumed, broken down by input/output. This is your cost tracking.
  • gen_ai.client.operation.duration: end-to-end latency per operation. This is your SLA.
None

There are more for streaming (time_to_first_chunk), tool duration, agent duration, and workflow duration, all following the same pattern.

One detail that matters: they're histograms by default, not averages. An average hides the request that took 30 seconds. A histogram gives you p50/p95/p99, which is where production problems actually live.

Events

Spans tell you how long something took, Metrics tell you how often, Events tell you what actually happened at a specific moment: the content, not the timing.

Three events matter:

  • gen_ai.client.inference.operation.details: the full message history, what was sent and what came back
  • gen_ai.evaluation.result: quality scores from evaluators
  • gen_ai.client.operation.exception: structured exception data when something fails
None

They're opt-in, and for a good reason: this is where sensitive data lives. Prompts contain user data, responses contain model output, exceptions contain internals. You enable events explicitly, only in the environments where capturing content is appropriate.

  • Spans: what happened in one request and how long each step took.
  • Metrics: aggregates across all requests, powering cost, latency, and percentiles.
  • Events: the actual content at a specific moment, opt-in because it's sensitive.
None

You can follow me on my social media:

👏 Clap if you enjoyed this article