Score traces

A score is a judgment about a trace or one of its spans: a name and a value, such as helpfulness 0.9, correct false or verdict good. A person scores from the dashboard, and code or an agent scores with an API key. Each score records who gave it.

One author has one score of a name per trace and span. Scoring the same name again replaces the value and keeps the score’s id, so a reviewer who changes their mind, or an evaluator that runs again, does not add a second score.

Before you start

You need an API key with traces:write to set and delete scores, and traces:read to read them. Keys from div login have both.

Score a trace

From the command line:

div score 4bf92f3577b34da6a3ce929d0e0e4736 helpfulness 0.9
div score 4bf92f3577b34da6a3ce929d0e0e4736 correct false --span 00f067aa0ba902b7
div score 4bf92f3577b34da6a3ce929d0e0e4736 verdict good --comment "cites the source"

From code, send the score to the API:

curl -X PUT https://api.divergentlabs.xyz/api/traces/4bf92f3577b34da6a3ce929d0e0e4736/scores \
  -H "Authorization: Bearer $DIVERGENT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "helpfulness", "value": 0.9, "comment": "clear and short"}'

The response is 201 for a new score and 200 for a replaced one, with the score in data.

Field	Description
name	Required. 1 to 64 letters, digits, dots, dashes or underscores.
value	Required. A number, true or false, or a string of up to 128 characters, which is a category.
span_id	A span of the trace to score instead of the whole trace.
comment	A note, up to 2,000 characters.
data_type	numeric, boolean or categorical. Optional; when given, it must match value.

A score needs its trace to exist. Score after your exporter has sent the trace’s spans: in a script, shut down the tracer provider first, which sends the last spans.

A trace keeps at most 200 scores.

Find traces by score

--score takes a name, a comparison and a value. A trace matches when any of its scores of that name passes:

div traces --score 'helpfulness<0.5' --since 7d
div traces --score 'verdict==good'
div traces --score 'correct==false' --errors

Numbers compare with >=, <=, >, <, == and !=. A category or a boolean compares with == and !=.

Keep a copy of every score

Code that keeps its own copy of the org’s scores, such as a job that builds a dataset from them, reads the scores once and then follows the feed of changes. The feed has every change in the order it happened, numbered with a seq:

op	Meaning
score_upserted	A score was set. score is the score as it is now.
score_deleted	A score was deleted. score_id names it.
trace_deleted	A trace was deleted, and every score of it with it.
trace_expired	A trace reached its 30 days, and every score of it went with it.

1. Read every page of GET /api/scores. Keep the snapshot_seq of the first page. Every page returns the same value.
2. Store each score by its id.
3. Read GET /api/scores/changes?after=<snapshot_seq> and apply each change: replace a score by id, remove one by id, or remove every score of a trace. Then read again with after set to the response’s next.
4. Keep next. To continue later, read the feed from it.

Changes made while you read the pages are in the feed after snapshot_seq. You may apply one that a page already showed; applying a change twice gives the same result.

div does all of this in one command, printing each score and then each change as one JSON object per line:

div scores --changes --resync --follow --json

When the command stops, it prints the seq to resume from: div scores --changes --after <seq> --follow --json.

The feed keeps 30 days of changes. A seq older than that is refused with cursor_expired. Start again from step 1, or run div scores --changes --resync.