# Compare model versions

> Publish model weights as versions, measure each version with the same evaluation, and rank them.

This guide shows how to measure several versions of a model with one evaluation script and rank them by a metric.

An evaluation has a name and one metric to rank by. Each `div eval run` measures one model version and adds it to the ranking.

## 1. Publish the weights as a version

Upload the weights to a volume, or write them there from a training run. Then publish them:

```bash
div storage put weights ./dist/llama-ft-v3 llama-ft/v3
div model publish llama-ft:v3 --volume weights --path llama-ft/v3
```

## 2. Write the evaluation script

The script loads the weights from `DIVERGENT_MODEL_PATH` and writes its score to `DIVERGENT_RESULT_PATH`:

```python
import json, os

model = load(os.environ["DIVERGENT_MODEL_PATH"])   # /mnt/weights/llama-ft/v3
accuracy = evaluate(model, held_out)

with open(os.environ["DIVERGENT_RESULT_PATH"], "w") as f:
    json.dump({"metrics": {"accuracy": accuracy}}, f)
```

## 3. Measure a version

The first run creates the evaluation, so it needs `--metric`:

```bash
div eval run held-out --model llama-ft:v3 --gpu h100 eval.py --metric metrics.accuracy
```

Measure more versions the same way. `--metric` is not needed after the first run:

```bash
div eval run held-out --model llama-ft:v4 --gpu h100 eval.py
```

For a metric where lower is better, such as loss, add `--lower` once.

A version that has already been measured with the same evaluation setup is not measured or billed again. `div eval run` prints the earlier score and exits `0`. Pass `--again` to measure it anyway.

## 4. Read the ranking

```bash
div eval report held-out
```

```text
1  llama-ft:v3  0.91  checks 1/1  2 runs  $0.84  run-k4t9
2  llama-ft:v2   0.8  checks 1/1  1 run   $0.42  run-p0q1
–  llama-ft:v1     –  checks 0/1  1 run   $0.40  run-a1b2  failed
held-out ranked by metrics.accuracy ↑ · suite 9f31c2ab · 4 runs · $1.66
```

## Keep versions comparable

Versions are ranked together only if they were measured with the same **suite**: the same evaluation code, command, image, `--env` values and checks. If you change any of these, the next run starts a new ranking. Earlier rankings are kept and shown separately in the report.

### Limit the suite to the evaluation's own files

By default, every uploaded file is part of the suite, so any change in the repository starts a new ranking. If the evaluation shares a repository with other code, name the evaluation's files with `--suite-path`:

```bash
div eval run ood --model llama-ft:v4 \
  --suite-path mask/eval --suite-path jobs/eval_ood.py \
  -- python jobs/eval_ood.py
```

The evaluation remembers the paths, so later runs do not need to repeat them. Include every file that affects the score, such as scoring code, helpers and data splits. To go back to all files, pass `--suite-path .`.

### Install a model's dependencies with `--setup`

If one model needs extra packages, install them with `--setup` instead of changing the command. `--setup` is not part of the suite, so the version is ranked with the others:

```bash
div eval run ood --model masker:v2 \
  --setup 'pip install transformers' -- python -m jobs.eval_ood
div eval run ood --model redact:v0.4.0 \
  --setup 'apt-get update && apt-get install -y curl' -- python -m jobs.eval_ood
```

`--setup` runs in `/workspace` before the command. If it fails, the run fails. Do not use it for anything that changes the score.

## Related

- [`div eval`](https://docs.divergentlabs.xyz/compute/cli/eval.md)
- [`div model`](https://docs.divergentlabs.xyz/compute/cli/model.md)