divergent/docs
ComputePrivate beta

Compare model versions

Publish model weights as versions, measure each version with the same evaluation, and rank them.

This guide shows how to measure several versions of a model with one evaluation script and rank them by a metric.

An evaluation has a name and one metric to rank by. Each div eval run measures one model version and adds it to the ranking.

1. Publish the weights as a version

Upload the weights to a volume, or write them there from a training run. Then publish them:

div storage put weights ./dist/llama-ft-v3 llama-ft/v3
div model publish llama-ft:v3 --volume weights --path llama-ft/v3

2. Write the evaluation script

The script loads the weights from DIVERGENT_MODEL_PATH and writes its score to DIVERGENT_RESULT_PATH:

import json, os

model = load(os.environ["DIVERGENT_MODEL_PATH"])   # /mnt/weights/llama-ft/v3
accuracy = evaluate(model, held_out)

with open(os.environ["DIVERGENT_RESULT_PATH"], "w") as f:
    json.dump({"metrics": {"accuracy": accuracy}}, f)

3. Measure a version

The first run creates the evaluation, so it needs --metric:

div eval run held-out --model llama-ft:v3 --gpu h100 eval.py --metric metrics.accuracy

Measure more versions the same way. --metric is not needed after the first run:

div eval run held-out --model llama-ft:v4 --gpu h100 eval.py

For a metric where lower is better, such as loss, add --lower once.

A version that has already been measured with the same evaluation setup is not measured or billed again. div eval run prints the earlier score and exits 0. Pass --again to measure it anyway.

4. Read the ranking

div eval report held-out
1  llama-ft:v3  0.91  checks 1/1  2 runs  $0.84  run-k4t9
2  llama-ft:v2   0.8  checks 1/1  1 run   $0.42  run-p0q1
–  llama-ft:v1     –  checks 0/1  1 run   $0.40  run-a1b2  failed
held-out ranked by metrics.accuracy ↑ · suite 9f31c2ab · 4 runs · $1.66

Keep versions comparable

Versions are ranked together only if they were measured with the same suite: the same evaluation code, command, image, --env values and checks. If you change any of these, the next run starts a new ranking. Earlier rankings are kept and shown separately in the report.

Limit the suite to the evaluation’s own files

By default, every uploaded file is part of the suite, so any change in the repository starts a new ranking. If the evaluation shares a repository with other code, name the evaluation’s files with --suite-path:

div eval run ood --model llama-ft:v4 \
  --suite-path mask/eval --suite-path jobs/eval_ood.py \
  -- python jobs/eval_ood.py

The evaluation remembers the paths, so later runs do not need to repeat them. Include every file that affects the score, such as scoring code, helpers and data splits. To go back to all files, pass --suite-path ..

Install a model’s dependencies with --setup

If one model needs extra packages, install them with --setup instead of changing the command. --setup is not part of the suite, so the version is ranked with the others:

div eval run ood --model masker:v2 \
  --setup 'pip install transformers' -- python -m jobs.eval_ood
div eval run ood --model redact:v0.4.0 \
  --setup 'apt-get update && apt-get install -y curl' -- python -m jobs.eval_ood

--setup runs in /workspace before the command. If it fails, the run fails. Do not use it for anything that changes the score.

esc