# Tracing

> Traces of your LLM calls from any OpenTelemetry exporter, with prompts, completions, tokens and scores.

Tracing keeps the traces your application sends with OpenTelemetry: each LLM call with its prompt, completion, model and tokens, each tool call with its arguments and result, and the agent steps around them. You read them in the dashboard, with [`div traces`](https://docs.divergentlabs.xyz/tracing/cli/traces.md), or from an agent with the [MCP tools](https://docs.divergentlabs.xyz/tracing/mcp.md), and you attach scores to them.

Tracing is in private beta and free while it is. An org is let in by request: open **Tracing** in the dashboard and request access.

## What tracing stores

Tracing stores what your exporter sends for 30 days, including prompts and completions. Before a span is stored, tracing removes:

- request and response headers recorded as attributes, except `content-type`, `content-length` and `user-agent`
- the value of any attribute whose name contains `authorization`, `cookie`, `api_key`, `api-key`, `secret`, `password`, `access_token`, `refresh_token` or `private_key`
- strings that look like API keys, bearer tokens, AWS access keys, GitHub and Slack tokens, or private keys, anywhere in the span, prompts included

This removal matches patterns and can miss a credential in an unusual format. To keep prompts and completions out of traces entirely, turn off content capture in your instrumentation. Each guide below shows how.

## Quickstart

This quickstart sends a trace from a Python script that calls an OpenAI-compatible API.

1. Create an API key with the `traces:write` scope under **Settings** in the dashboard, or run `div login`. Keys from `div login` have `traces:read` and `traces:write`.

2. Install the OpenTelemetry SDK, the OTLP exporter and the OpenAI instrumentation.

   ```bash
   pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http opentelemetry-instrumentation-openai-v2 openai
   ```

3. Point the exporter at tracing. `%20` is the space after `Bearer`: the variable is URL-encoded.

   ```bash
   export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.divergentlabs.xyz/api/otlp
   export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20$DIVERGENT_API_KEY"
   export OTEL_SERVICE_NAME=my-app
   export OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
   export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY
   ```

4. Run a script that sets up tracing and makes a call.

   ```python
   from opentelemetry import trace
   from opentelemetry.sdk.trace import TracerProvider
   from opentelemetry.sdk.trace.export import BatchSpanProcessor
   from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
   from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor
   import openai

   provider = TracerProvider()
   provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
   trace.set_tracer_provider(provider)
   OpenAIInstrumentor().instrument()

   client = openai.OpenAI()
   client.chat.completions.create(model="gpt-4o-mini", messages=[{"role": "user", "content": "What is 2+2?"}])
   provider.shutdown()
   ```

5. List the trace and open it.

   ```bash
   div traces --since 10m
   div trace <trace-id>
   ```

   The output looks like this:

   ```text
   chat gpt-4o-mini  1 span  412ms  21→7 tokens
   0µs  412ms  generation  chat gpt-4o-mini  gpt-4o-mini-2024-07-18  21→7  5cb3760d6ed6d813
   ```

## Libraries tracing reads

Tracing accepts any OTLP/HTTP trace export. It reads the model, tokens, messages, session and user from spans written by these libraries:

| Library | Guide |
| --- | --- |
| OpenTelemetry GenAI instrumentations, such as `opentelemetry-instrumentation-openai-v2` | [Send traces from Python](https://docs.divergentlabs.xyz/tracing/guides/opentelemetry-python.md) |
| Vercel AI SDK 5, 6 and 7 | [Send traces from the AI SDK](https://docs.divergentlabs.xyz/tracing/guides/vercel-ai-sdk.md) |
| OpenInference | [Send traces with OpenInference](https://docs.divergentlabs.xyz/tracing/guides/openinference.md) |
| OpenLLMetry (Traceloop) | [Send traces with OpenLLMetry](https://docs.divergentlabs.xyz/tracing/guides/openllmetry.md) |

Spans from other libraries are stored with their attributes, and show no model, tokens or messages. [OTLP endpoint](https://docs.divergentlabs.xyz/tracing/otlp.md#attributes-tracing-reads) lists the attributes tracing reads.

## Limits

| Limit | Value |
| --- | --- |
| Retention | 30 days from when tracing first receives a trace's span |
| Spans per org per month | 1,000,000 |
| Stored span data per org per month | 5 GiB |
| Spans per trace | 10,000 |
| Input and output per span | 128 KiB each. Longer content is cut, and the span says so. |
| Request body | 4 MiB as sent, 16 MiB uncompressed |

Spans over a monthly limit are refused until the first day of the next month, UTC. Your exporter logs how many spans were refused and why. `div traces` and the dashboard show this month's usage.

## Times

A span's start and end are the times your application reported. Tracing orders traces, and filters them with `--since` and `--until`, by when it first received a span of the trace. A trace that arrives late, or from a host with a wrong clock, is listed when it arrived.