divergent/docs
Compute · Commands

div eval

Measure a model version with an evaluation, list evaluations, and print an evaluation's ranking.

Synopsis

div eval run <evaluation> --model <model:tag> [flags] <entrypoint>
div eval ls [--project <name>]
div eval report <evaluation> [--project <name>]

Measures model versions and ranks them by one metric from each run’s result.

Subcommands

Subcommand Description
run Start a run that measures one model version for the evaluation. Creates the evaluation on first use.
ls List the org’s evaluations, each with its leading version.
report Print the evaluation’s versions, best first.

Arguments

Argument Description
<evaluation> Evaluation name, such as held-out.
<entrypoint> What to run, as for div run.

Flags for div eval run

Flag Description Default
--model <model:tag> Model version to measure, such as llama-ft:v3, or a version ID. Required.
--metric <path> Result path to rank by, such as metrics.accuracy. Required when the evaluation is created. Passing it later changes the metric.
--lower Rank lower values first, for example for loss. Saved with the evaluation. higher is better
--higher Rank higher values first.
--again Measure the version even if this suite has already measured it.
--suite-path <path> File or directory that makes up the evaluation’s code. Repeatable. Saved with the evaluation. . resets it to all uploaded files. all uploaded files
--setup <command> Command to run in /workspace before the entrypoint, such as 'pip install transformers'. Not part of the suite.

div eval run also accepts every div run flag, including --gpu, --check, --max-spend, --env, --detach and --json.

Flags for div eval ls and div eval report

Flag Description Default
--project <name> Only look in this project. all projects
--json Print as JSON. report --json includes earlier suites.
--token <key> API key to use. see Configuration
--api-url <url> API URL to use. see Configuration

Environment variables in an eval run

In addition to the variables every run has (see Runtime environment):

Variable Value
DIVERGENT_MODEL the version, such as llama-ft:v3
DIVERGENT_MODEL_VERSION_ID the version’s ID
DIVERGENT_MODEL_PATH where the weights are mounted, such as /mnt/weights/llama-ft/v3, if the version has a path
DIVERGENT_MODEL_DIGEST a fingerprint of the version’s weights
DIVERGENT_EVALUATION the evaluation’s name

Suites

A suite is the combination of evaluation code, entrypoint, image, --env values and checks. div eval report ranks only runs from the current suite. Earlier suites are listed separately.

  • The code is identified by file contents and names, not by upload location. Uploading the same files again gives the same suite.
  • With --suite-path, only files under those paths count as code. Changing the paths starts a new suite. A path that matches no uploaded file is an error.
  • --setup is never part of the suite.

If this suite has already measured the version, div eval run starts nothing, prints the earlier value and exits 0. If a run for the version is still in progress, div eval run follows it. A failed run does not count as a measurement.

Output

div eval run prints the same output as div run.

div eval report prints one row per version to stdout and a summary line to stderr:

1  llama-ft:v3  0.91  checks 1/1  2 runs  $0.84  run-k4t9
2  llama-ft:v2   0.8  checks 1/1  1 run   $0.42  run-p0q1
–  llama-ft:v1     –  checks 0/1  1 run   $0.40  run-a1b2  failed
held-out ranked by metrics.accuracy ↑ · suite 9f31c2ab · 4 runs · $1.66

Columns: rank, version, metric value from the most recent run that produced one, checks passed, number of runs, total cost, latest run, and status if it did not succeed. A version with no value is listed last with –.

Exit codes

Code Meaning
the run’s exit code div eval run: the run ended with a non-zero exit code.
0 Success, or the version was already measured.
1 The API rejected the request, or the run stopped without an exit code.
2 Invalid flags or arguments, a new evaluation without --metric, or a --suite-path that matches nothing.

Examples

div eval run held-out --model llama-ft:v3 eval.py --metric metrics.accuracy
div eval run held-out --model llama-ft:v4 eval.py --check 'metrics.accuracy>=0.8'
div eval run ood --model llama-ft:v4 --suite-path mask/eval -- python mask/eval/ood.py
div eval run ood --model redact:v0.4.0 --setup 'apt-get install -y curl' -- python -m jobs.eval_ood
div eval report held-out
div eval ls

See Compare model versions.

esc