# div eval > Measures a model version and ranks it against the others. A version already measured under the same suite is not measured, or billed, again. ```bash div eval run held-out --model llama-ft:v3 eval.py --metric metrics.accuracy div eval run held-out --model llama-ft:v4 eval.py --check 'metrics.accuracy>=0.8' div eval report held-out div eval ls ``` An evaluation is a name and the one number it ranks by. `div eval run` measures one model version for it: it is [`div run`](https://docs.divergentlabs.xyz/compute/cli/run.md) — the directory shipped, the run placed, the log followed, the exit code the run's own — with two more things in the run's environment and a place for the number to go. ## What the run is told | Variable | What it holds | | --- | --- | | `DIVERGENT_MODEL` | the version, as `llama-ft:v3` | | `DIVERGENT_MODEL_PATH` | where its weights are mounted, like `/mnt/weights/llama-ft/v3`, when the version says | | `DIVERGENT_MODEL_DIGEST` | the version's digest | | `DIVERGENT_EVALUATION` | the evaluation's name | | `DIVERGENT_RESULT_PATH` | where to write what it measured | What it writes to `DIVERGENT_RESULT_PATH` is its [result](https://docs.divergentlabs.xyz/compute/cli/result.md), and the evaluation ranks versions by `--metric`, a path into it: ```python import json, os model = load(os.environ["DIVERGENT_MODEL_PATH"]) accuracy = evaluate(model, held_out) with open(os.environ["DIVERGENT_RESULT_PATH"], "w") as f: json.dump({"metrics": {"accuracy": accuracy}}, f) ``` The run grades nothing itself. The control plane reads the number from the result once the run has exited, and judges any `--check` against it there. ## Flags | Flag | What it does | | --- | --- | | `--model ` | the version to measure; a version id works too | | `--metric ` | what the evaluation ranks by, like `metrics.accuracy`. Needed the first time, when the evaluation is made; changes it after that | | `--lower` | lower is better, for a loss. `--higher` puts it back | | `--again` | measure it even though this suite already has | | `--suite-path ` | a file or directory that is the eval's code, once per path. The evaluation keeps them; `--suite-path .` goes back to everything shipped. See [the eval's own code](#the-evals-own-code) | | `--setup ` | run first, in the shipped code: what this model's runtime needs, like `'apt-get install -y curl'`. Never part of the suite. See [what one model needs](#what-one-model-needs) | Every flag [`div run`](https://docs.divergentlabs.xyz/compute/cli/run.md) takes works here too: `--gpu`, `--check`, `--max-spend`, `--env`, `--detach`, `--json` and the rest. ## One suite at a time The number a run produces depends on more than the model: the code, the entrypoint, the image, the environment and the checks all decide it. Together they are the run's **suite**, and `div eval report` only ranks runs of one suite against each other. The code is known by what it holds — a digest of every file shipped and its name — not by where it was put, so shipping the same directory twice is the same suite. Change any of them and the next run starts a new ranking. The runs of the old suite are kept and shown apart, never ranked against the new ones: a number measured one way next to a number measured another says nothing about the models. ### The eval's own code When the eval lives in a repository with the code it measures, every file shipped is too much: a commit to the training code starts a new ranking, and the version at the top has to be measured again before anything can be compared with it. Say which paths are the eval's code, and only the files under them are digested: ```sh div eval run ood --model llama-ft:v4 \ --suite-path mask/eval --suite-path jobs/eval_ood.py \ python jobs/eval_ood.py ``` The whole directory is still shipped, so the eval can import from anywhere in it; only what counts as the suite narrows. The evaluation keeps the paths, so later runs need not repeat them, and `div eval report` names them beside the suite. The paths are part of the suite too: changing them starts a new ranking, and so does setting them the first time. A path that matches nothing shipped exits 2 before anything is uploaded. A file the eval reads that is outside its paths is not part of the suite, so a change to it is ranked as if nothing had changed. Name everything that decides the number: the scoring code, its helpers, and the data split, if it is shipped. ### What one model needs Models do not all load the same way. One is PyTorch and needs `transformers`; a competitor's ships a native library that loads libcurl, which the image does not have. Put that in the command and it becomes part of the suite, so adding the one model starts a new ranking, and every version already ranked has to be measured again to be compared with it. Put it in `--setup` instead: ```sh div eval run ood --model masker:v2 \ --setup 'pip install transformers' -- python -m jobs.eval_ood div eval run ood --model redact:v0.4.0 \ --setup 'apt-get update && apt-get install -y curl' -- python -m jobs.eval_ood ``` Both land in the same suite, ranked together. The container runs the setup in `/workspace`, after the code is copied there and before the command, so `pip install -r requirements.txt` finds its file. A setup that fails fails the run, which then measured nothing. `--setup` is for what a model needs to load, not for how it is scored. Anything that changes the number — a different tokenizer for the scorer, a pinned version of the metric library — belongs in the command or the shipped code, where it is part of the suite. The run keeps its setup, the run's page shows it, and "same run from the CLI" repeats it. ## A version is measured once A version this suite has already measured is not measured again. `div eval run` prints the number it got, places nothing, and exits 0, so running a sweep a second time costs only the versions that are new: ```text llama-ft:v3 was measured under this suite by run-k4t9: metrics.accuracy 0.91 nothing placed. --again measures it again ``` A version this suite is measuring right now is followed rather than placed twice. A run that failed measured nothing, so its version is measured again. `--again` measures anyway, when the suite is stochastic and you want another draw. ## The report ```text $ div eval report held-out 1 llama-ft:v3 0.91 checks 1/1 2 runs $0.84 run-k4t9 2 llama-ft:v2 0.8 checks 1/1 1 run $0.42 run-p0q1 – llama-ft:v1 – checks 0/1 1 run $0.40 run-a1b2 failed held-out ranked by metrics.accuracy ↑ · suite 9f31c2ab · 4 runs · $1.66 ``` One row per version, from its most recent run that produced a number, best first. A version whose runs produced no number is last, with `–`, never ranked as zero. A run that missed a [check](https://docs.divergentlabs.xyz/compute/cli/run.md#checks) is still a good run that measured the number it missed with, so it is ranked, and its checks column says it missed. That is the only place the verdict shows: the run's own status is its exit code's, and its page has the evaluation in its Eval tab. The rows go to stdout; the line about the suite goes to stderr. `--json` prints the whole report, earlier suites included. `div eval ls` lists the org's evaluations, each with the version that leads it. ## Versions A version is weights on a volume, published with [`div model publish`](https://docs.divergentlabs.xyz/compute/cli/model.md): ```bash div storage put weights ./dist/llama-ft-v3 llama-ft/v3 div model publish llama-ft:v3 --volume weights --path llama-ft/v3 --run run-k4t9 ``` An eval run of that version is told `DIVERGENT_MODEL_PATH=/mnt/weights/llama-ft/v3`, since every run mounts every volume of its org.