divergent/docs
ComputePrivate beta

Report a run's metrics and check them

Write a run's metrics to its result file, pass or fail the run's checks against them, and read them back from a script.

This guide shows how to make a run report what it measured, and how to check that number automatically.

An exit code tells you whether a run crashed. A result tells you what the run measured. A check compares the result to a target.

1. Write the result

Every run has the environment variable DIVERGENT_RESULT_PATH. Before the run exits, write one JSON object to that path:

import json, os

with open(os.environ["DIVERGENT_RESULT_PATH"], "w") as f:
    json.dump({"metrics": {"accuracy": 0.91, "loss": 0.42}}, f)

You can rewrite the file as often as you like. The version on disk when the process exits is the one kept. The object can be at most 64 KiB.

2. Add checks

Pass --check once per condition:

div run eval.py --check 'metrics.accuracy>=0.8' --check 'metrics.loss<0.6'

When the run exits, each check is evaluated against the result and the verdict is written to the log:

check passed: metrics.accuracy >= 0.8 (found 0.91)
check failed: metrics.loss < 0.6 (found 0.71)

If the result has no value at a check’s path, the verdict is unknown, which does not count as a pass. See Check syntax.

A missed check does not change the run’s status or exit code. A run that exits 0 and misses a check is still succeeded, and the last line of div run reports the missed checks. To fail a script on a missed check, read the verdicts from --json output:

div run --json eval.py --check 'metrics.accuracy>=0.8' > run.json
jq -e 'all(.checks[]; .verdict == "passed")' run.json > /dev/null || exit 1

3. Read the result

div result run-7                      # the whole object
div result run-7 metrics.accuracy     # one value: 0.91

div result exits 1 if the run has not finished, wrote no result, or has no value at the path. Use this to guard a script:

acc=$(div result run-7 metrics.accuracy) || exit 1
esc