Chart training metrics

This guide shows how to record metrics while a run trains, so they appear as charts on the run’s Telemetry tab in the dashboard and in div telemetry.

1. Append metrics to the telemetry file

Every run has the environment variable DIVERGENT_TELEMETRY_PATH. Append one JSON object per line:

import json, os

def record(**values):
    with open(os.environ["DIVERGENT_TELEMETRY_PATH"], "a") as f:
        print(json.dumps(values), file=f, flush=True)

record(step=120, loss=0.41, lr=3e-4)

Each line becomes one point. Nested objects become dotted keys: {"eval": {"loss": 0.4}} is eval.loss. Non-numeric values are ignored. Points appear about one second after they are written.

Numbers printed to the log are not charted. Only the telemetry file is read.

With a Hugging Face Trainer

Add a callback that writes each log entry to the file:

import json, os
from transformers import TrainerCallback

class Telemetry(TrainerCallback):
    def on_log(self, args, state, control, logs=None, **kwargs):
        if state.is_world_process_zero and logs:
            with open(os.environ["DIVERGENT_TELEMETRY_PATH"], "a") as f:
                f.write(json.dumps({"step": state.global_step, **logs}) + "\n")

trainer = Trainer(model=model, args=args, callbacks=[Telemetry()], ...)

2. Lay out the charts (optional)

Without a layout, the Telemetry tab shows one chart per key, plotted against step when the run records it. To choose the charts, add a telemetry.toml to the root of the directory you run from:

x = "step"

[[panel]]
title = "Loss"
y = ["loss", "eval_loss"]
scale = "log"

[[stat]]
title = "Samples/s"
y = "eval_samples_per_second"

The layout is saved with the run when it starts. If the file is invalid, the run is rejected and the error names the table and key to fix. See telemetry.toml for every key.

3. Read the values

div telemetry run-7                        # every key, with count, last, min and max
div telemetry run-7 loss eval_loss         # tab-separated columns
div telemetry run-7 loss -f | tail -1      # follow while the run trains
div telemetry run-7 --csv > run-7.csv      # every key, as CSV

Telemetry is for charts only. It does not affect the run’s status or its checks.

Related

• div telemetry, including the limits on lines, keys and points