divergent/docs
TracingPrivate beta

Tracing

Traces of your LLM calls from any OpenTelemetry exporter, with prompts, completions, tokens and scores.

Tracing keeps the traces your application sends with OpenTelemetry: each LLM call with its prompt, completion, model and tokens, each tool call with its arguments and result, and the agent steps around them. You read them in the dashboard, with div traces, or from an agent with the MCP tools, and you attach scores to them.

Tracing is in private beta and free while it is. An org is let in by request: open Tracing in the dashboard and request access.

What tracing stores

Tracing stores what your exporter sends for 30 days, including prompts and completions. Before a span is stored, tracing removes:

  • request and response headers recorded as attributes, except content-type, content-length and user-agent
  • the value of any attribute whose name contains authorization, cookie, api_key, api-key, secret, password, access_token, refresh_token or private_key
  • strings that look like API keys, bearer tokens, AWS access keys, GitHub and Slack tokens, or private keys, anywhere in the span, prompts included

This removal matches patterns and can miss a credential in an unusual format. To keep prompts and completions out of traces entirely, turn off content capture in your instrumentation. Each guide below shows how.

Quickstart

This quickstart sends a trace from a Python script that calls an OpenAI-compatible API.

  1. Create an API key with the traces:write scope under Settings in the dashboard, or run div login. Keys from div login have traces:read and traces:write.

  2. Install the OpenTelemetry SDK, the OTLP exporter and the OpenAI instrumentation.

    pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http opentelemetry-instrumentation-openai-v2 openai

  3. Point the exporter at tracing. %20 is the space after Bearer: the variable is URL-encoded.

    export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.divergentlabs.xyz/api/otlp
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20$DIVERGENT_API_KEY"
export OTEL_SERVICE_NAME=my-app
export OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY

  4. Run a script that sets up tracing and makes a call.

    from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor
import openai

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
OpenAIInstrumentor().instrument()

client = openai.OpenAI()
client.chat.completions.create(model="gpt-4o-mini", messages=[{"role": "user", "content": "What is 2+2?"}])
provider.shutdown()

  5. List the trace and open it.

    div traces --since 10m
div trace <trace-id>

    The output looks like this:

    chat gpt-4o-mini  1 span  412ms  21→7 tokens
0µs  412ms  generation  chat gpt-4o-mini  gpt-4o-mini-2024-07-18  21→7  5cb3760d6ed6d813

Libraries tracing reads

Tracing accepts any OTLP/HTTP trace export. It reads the model, tokens, messages, session and user from spans written by these libraries:

Library Guide
OpenTelemetry GenAI instrumentations, such as opentelemetry-instrumentation-openai-v2 Send traces from Python
Vercel AI SDK 5, 6 and 7 Send traces from the AI SDK
OpenInference Send traces with OpenInference
OpenLLMetry (Traceloop) Send traces with OpenLLMetry

Spans from other libraries are stored with their attributes, and show no model, tokens or messages. OTLP endpoint lists the attributes tracing reads.

Limits

Limit Value
Retention 30 days from when tracing first receives a trace’s span
Spans per org per month 1,000,000
Stored span data per org per month 5 GiB
Spans per trace 10,000
Input and output per span 128 KiB each. Longer content is cut, and the span says so.
Request body 4 MiB as sent, 16 MiB uncompressed

Spans over a monthly limit are refused until the first day of the next month, UTC. Your exporter logs how many spans were refused and why. div traces and the dashboard show this month’s usage.

Times

A span’s start and end are the times your application reported. Tracing orders traces, and filters them with --since and --until, by when it first received a span of the trace. A trace that arrives late, or from a host with a wrong clock, is listed when it arrived.

esc