---
title: "LLM Observability"
summary: "Sends agent runs to an LLM observability platform for span-level traces of each LLM call and tool use, plus aggregate cost, latency, and success metrics"
status: "emerging"
category: "Reliability & Eval"
url: "https://agentic-patterns.com/patterns/llm-observability"
source: "https://lethain.com/agents-logging/"
authors: ["Nikola Balic (@nibzard)"]
tags: ["observability","logging","debugging","tracing","datadog","langsmith","spans","monitoring","llmops"]
signals: ["Multi-step agent workflows are hard to debug from raw logs","Non-engineers need to inspect workflow runs","Team needs cost, latency, and success metrics across runs"]
anti_signals: ["Simple, deterministic tools with no agent behavior","Single-step operations where standard logs suffice","Budget does not cover observability spend"]
updated_at: "2026-10-13"
---

# LLM Observability

> Sends agent runs to an LLM observability platform for span-level traces of each LLM call and tool use, plus aggregate cost, latency, and success metrics

## Use when

- Multi-step agent workflows are hard to debug from raw logs
- Non-engineers need to inspect workflow runs
- Team needs cost, latency, and success metrics across runs

## Avoid when

- Simple, deterministic tools with no agent behavior
- Single-step operations where standard logs suffice
- Budget does not cover observability spend

## Problem

Agents introduce **non-determinism**—the same input can produce different outputs. When agents do something sub-optimal, users flag it as a "bug" even if it's just prompt ambiguity. Debugging these issues requires tracing through complex multi-step workflows, but standard logging (CloudWatch, Lambda logs) is painful to navigate. Engineers need easy access to workflow execution details to debug quickly, or agents won't get adopted.

## Solution

Integrate **LLM observability platforms** (Datadog LLM Observability, LangSmith, etc.) that provide **span-level tracing** of agent workflows. Instead of spelunking through raw logs, get a visual UI showing each step of the agent's execution.

**Key capabilities:**

- **Span visualization**: See each LLM call, tool use, and intermediate result
- **Workflow linking**: Trace from user input through all sub-steps to final output
- **Dashboarding**: Aggregate metrics on cost, latency, success rates
- **Accessible debugging**: Non-engineers can inspect runs without raw log access (in the source deployment, Datadog access was still limited to the technology organization)
- **Event sourcing**: Structured logs enable replay, state reconstruction, and verification
- **Vendor-neutral tracing**: OpenTelemetry GenAI semantic conventions and OpenInference define portable span attributes, so traces are not locked to one platform

**Evolution from standard logging:**

1. **Print to stdout** → Captured in Lambda logs (hard to access)
2. **Slack channel** → Post run summary + AWS log link (better, still spelunking)
3. **LLM observability** → Visual span tracing, easy navigation

```mermaid
graph LR
    A[Agent Run] --> B[Observability SDK]
    B --> C[Span: LLM Call 1]
    B --> D[Span: Tool Use]
    B --> E[Span: LLM Call 2]
    C --> F[Trace UI]
    D --> F
    E --> F
    F --> G[Debug View]

    style F fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
```

## How to use it

**Integration steps:**

1. **Choose observability platform**: Datadog LLM Observability, LangSmith, Arize, etc.
2. **Install SDK**: Add to your agent code (often 1-2 lines)
3. **Configure tracing**: Tag workflows, users, environments
4. **Access controls**: Give broader team access (not just engineers)

**Example with Datadog:**

```python
from ddtrace import tracer

# Automatically traces LLM calls
@tracer.wrap("agent_workflow")
def run_agent(query):
    result = agent.run(query)
    # Each LLM call, tool use becomes a span
    return result
```

**Best practices:**

- **Tag everything**: Workflow name, user ID, environment (prod/staging)
- **Link to dashboards**: Make observability UI discoverable from chat
- **Share access**: Don't restrict to eng org; workflow creators need visibility
- **Monitor aggregate metrics**: Track success rates, latency, costs over time
- **Use structured formats**: JSONL with schema versioning for long-running systems
- **Aggregate at write time**: Log enrichment happens during capture, not during queries

## Trade-offs

**Pros:**

- **Fast debugging**: Navigate complex workflows visually
- **Accessible**: Broader teams can debug without log permissions; the source deployment still limited access to the technology organization
- **Aggregated metrics**: See patterns across many runs
- **Span-level detail**: Drill into any step of execution

**Cons:**

- **Vendor dependency**: Locked into observability platform
- **Cost**: Enterprise observability can be expensive
- **Overhead**: Adds latency (usually minimal)
- **Access control**: Balancing visibility with security

**When NOT to use:**

- Simple, deterministic tools (no agent behavior)
- Single-step operations (standard logs suffice)
- Budget constraints preventing observability spend

## References

* [Building an internal agent: Logging and debugability](https://lethain.com/agents-logging/) - Will Larson (Imprint, 2025)
* [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://arxiv.org/abs/2503.11926) - OpenAI (March 2025); the reasoning-monitoring evidence behind CoT observability
* [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai) - vendor-neutral span attributes for LLM and agent calls (in development)
* [OpenInference](https://github.com/Arize-ai/openinference) - open instrumentation standard for LLM/agent tracing
* [ESAA: Event Sourcing for Autonomous Agents](https://arxiv.org/abs/2602.23193v1) - Elzo Brito dos Santos Filho et al. (February 2026)
* [Chain of Thought Monitorability: A Fragile Opportunity](https://arxiv.org/abs/2510.19476) - Korbak et al. (October 2025)
* Datadog LLM Observability documentation
* LangSmith documentation
* Related: Agent-First Tooling and Logging, Chain-of-Thought Monitoring & Interruption