# Score traces

> Attach scores to traces from code, the CLI or an agent, find traces by score, and keep a copy of every score in sync.

A score is a judgment about a trace or one of its spans: a name and a value, such as `helpfulness` `0.9`, `correct` `false` or `verdict` `good`. A person scores from the dashboard, and code or an agent scores with an API key. Each score records who gave it.

One author has one score of a name per trace and span. Scoring the same name again replaces the value and keeps the score's id, so a reviewer who changes their mind, or an evaluator that runs again, does not add a second score.

## Before you start

You need an API key with `traces:write` to set and delete scores, and `traces:read` to read them. Keys from `div login` have both.

## Score a trace

From the command line:

```bash
div score 4bf92f3577b34da6a3ce929d0e0e4736 helpfulness 0.9
div score 4bf92f3577b34da6a3ce929d0e0e4736 correct false --span 00f067aa0ba902b7
div score 4bf92f3577b34da6a3ce929d0e0e4736 verdict good --comment "cites the source"
```

From code, send the score to the API:

```bash
curl -X PUT https://api.divergentlabs.xyz/api/traces/4bf92f3577b34da6a3ce929d0e0e4736/scores \
  -H "Authorization: Bearer $DIVERGENT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "helpfulness", "value": 0.9, "comment": "clear and short"}'
```

The response is `201` for a new score and `200` for a replaced one, with the score in `data`.

| Field | Description |
| --- | --- |
| `name` | Required. 1 to 64 letters, digits, dots, dashes or underscores. |
| `value` | Required. A number, `true` or `false`, or a string of up to 128 characters, which is a category. |
| `span_id` | A span of the trace to score instead of the whole trace. |
| `comment` | A note, up to 2,000 characters. |
| `data_type` | `numeric`, `boolean` or `categorical`. Optional; when given, it must match `value`. |

A score needs its trace to exist. Score after your exporter has sent the trace's spans: in a script, shut down the tracer provider first, which sends the last spans.

A trace keeps at most 200 scores.

## Find traces by score

`--score` takes a name, a comparison and a value. A trace matches when any of its scores of that name passes:

```bash
div traces --score 'helpfulness<0.5' --since 7d
div traces --score 'verdict==good'
div traces --score 'correct==false' --errors
```

Numbers compare with `>=`, `<=`, `>`, `<`, `==` and `!=`. A category or a boolean compares with `==` and `!=`.

## Keep a copy of every score

Code that keeps its own copy of the org's scores, such as a job that builds a dataset from them, reads the scores once and then follows the feed of changes. The feed has every change in the order it happened, numbered with a `seq`:

| `op` | Meaning |
| --- | --- |
| `score_upserted` | A score was set. `score` is the score as it is now. |
| `score_deleted` | A score was deleted. `score_id` names it. |
| `trace_deleted` | A trace was deleted, and every score of it with it. |
| `trace_expired` | A trace reached its 30 days, and every score of it went with it. |

1. Read every page of `GET /api/scores`. Keep the `snapshot_seq` of the first page. Every page returns the same value.
2. Store each score by its `id`.
3. Read `GET /api/scores/changes?after=<snapshot_seq>` and apply each change: replace a score by `id`, remove one by `id`, or remove every score of a trace. Then read again with `after` set to the response's `next`.
4. Keep `next`. To continue later, read the feed from it.

Changes made while you read the pages are in the feed after `snapshot_seq`. You may apply one that a page already showed; applying a change twice gives the same result.

`div` does all of this in one command, printing each score and then each change as one JSON object per line:

```bash
div scores --changes --resync --follow --json
```

When the command stops, it prints the `seq` to resume from: `div scores --changes --after <seq> --follow --json`.

The feed keeps 30 days of changes. A `seq` older than that is refused with `cursor_expired`. Start again from step 1, or run `div scores --changes --resync`.