div telemetry

div telemetry run-7
div telemetry run-7 loss eval_loss
div telemetry run-7 loss -f | tail -1

A run’s telemetry is the numbers it writes while it runs: loss by step, learning rate, throughput. They are drawn on the run’s Telemetry tab, and div telemetry prints them.

Writing it

Every run is given DIVERGENT_TELEMETRY_PATH. Append one JSON object per line there, whenever there is something to say:

import json, os

with open(os.environ["DIVERGENT_TELEMETRY_PATH"], "a") as f:
    print(json.dumps({"step": 120, "loss": 0.41, "lr": 3e-4}), file=f, flush=True)

Each line becomes a point. Every number in it is kept under its key, with nested objects flattened into dotted keys: {"eval": {"loss": 0.4}} is eval.loss. Strings, booleans and nulls are passed over, and so are NaN and Infinity, which Python’s json writes and JSON does not have. The rest of the line is still kept. Lines arrive about a second after they are written.

Only this file is read. A number printed to the log stays text: nothing parses the log. Telemetry is drawn and never judged either: a run’s checks read its result, and nothing a point says changes how a run ends.

From a Hugging Face Trainer

The Trainer prints {'eval_loss': 0.0267, 'epoch': 0.04} to the log, and those numbers stay text there. A callback writes the same dict to the file:

import json, os
from transformers import TrainerCallback

class Telemetry(TrainerCallback):
    def on_log(self, args, state, control, logs=None, **kwargs):
        if state.is_world_process_zero and logs:
            with open(os.environ["DIVERGENT_TELEMETRY_PATH"], "a") as f:
                f.write(json.dumps({"step": state.global_step, **logs}) + "\n")

trainer = Trainer(model=model, args=args, callbacks=[Telemetry()], ...)

Laying it out

A telemetry.toml at the top of the directory div run ships lays out the Telemetry tab. --telemetry <file> names another. It is read when the run is created and kept with it, so a run shows what it was launched with.

x = "step"                       # what every panel is drawn against

[[panel]]
title = "Loss"
y = ["loss", "eval_loss"]        # up to 4 keys on one panel
scale = "log"

[[panel]]
title = "F1"
y = ["eval_strict_f1", "eval_typed_f1"]
min = 0.9
max = 1.0
rule = 0.95                      # one horizontal line, like a check's threshold

[[panel]]
y = "eval_leak_recall"
wide = true                      # spans both columns

[[stat]]
title = "Samples/s"
y = "eval_samples_per_second"
reduce = "last"                  # last, min or max

Key	In	Means	When left out
x	the file, a panel	the key a panel is drawn against	the point’s place in the run
title	a panel, a stat	what it is called	its keys
y	a panel, a stat	the key, or keys, to draw	required
scale	a panel	linear or log	linear
min, max	a panel	the y axis’s bounds	fitted to the points
rule	a panel	a horizontal reference line	none
wide	a panel	span both columns	false
reduce	a stat	last, min or max	last

A point without a panel’s x key has nowhere to go on that panel, so it is left off. A key no point has written yet shows as “no points yet”, never as a flat line at zero.

A run without a telemetry.toml gets one panel per key, in the order the keys first arrived, drawn against step when the run writes one. A file that cannot be read refuses the run, the way a check that cannot be read does. That covers a key misspelt, a scale that does not exist, or a min above the max, and the error says which table and which key:

div: the API refused the run: telemetry [[panel]] 2 has "scael", which is not one of title, x, y, scale, min, max, rule, wide

Reading it

With no keys, div telemetry prints every key the run has written, with how many points carry it and the last, lowest and highest value:

key        count  last     min      max
step       412    41200    100      41200
loss       412    0.0412   0.0398   2.31
eval_loss  41     0.0267   0.0261   1.88

With keys, it prints the points as tab-separated columns on stdout: x first, then one column per key. A point missing a key has an empty cell there, never a zero:

div telemetry run-7 loss eval_loss > curve.tsv
div telemetry run-7 loss --x epoch
div telemetry run-7 --csv > run-7.csv

Flag	Does
--x <key>	the first column; step when the run writes one, the point’s place in the run otherwise
--csv	the same table with commas, quoted where a cell needs it; with no keys, a column for every key the run has written
--json	every point as one JSON object per line, {"seq": 1, "data": {...}}
-f, --follow	keeps printing points as they arrive, until the run ends

What is not kept

A line is dropped whole when it is not a JSON object, is over 4 KiB, or has more than 64 numbers. A key that is not letters, digits, _ and - between dots, like train/loss, is left out of its point. A run keeps at most 256 keys and 200,000 points.

What is dropped is counted. The log says so the first time each reason comes up, and div telemetry repeats the counts on stderr:

telemetry: dropped a line that is not JSON; write one JSON object per line

Any value of one of the org’s secrets is masked before a line is read, as it is in the log.