# div telemetry

> Prints the numbers a run wrote to DIVERGENT_TELEMETRY_PATH as it ran, and how its telemetry.toml lays them out on the run's Telemetry tab.

```bash
div telemetry run-7
div telemetry run-7 loss eval_loss
div telemetry run-7 loss -f | tail -1
```

A run's telemetry is the numbers it writes while it runs: loss by step, learning rate, throughput. They are drawn on the run's Telemetry tab, and `div telemetry` prints them.

## Writing it

Every run is given `DIVERGENT_TELEMETRY_PATH`. Append one JSON object per line there, whenever there is something to say:

```python
import json, os

with open(os.environ["DIVERGENT_TELEMETRY_PATH"], "a") as f:
    print(json.dumps({"step": 120, "loss": 0.41, "lr": 3e-4}), file=f, flush=True)
```

Each line becomes a point. Every number in it is kept under its key, with nested objects flattened into dotted keys: `{"eval": {"loss": 0.4}}` is `eval.loss`. Strings, booleans and nulls are passed over, and so are `NaN` and `Infinity`, which Python's `json` writes and JSON does not have. The rest of the line is still kept. Lines arrive about a second after they are written.

Only this file is read. A number printed to the log stays text: nothing parses the log. Telemetry is drawn and never judged either: a run's [checks](https://docs.divergentlabs.xyz/compute/cli/run.md#checks) read its [result](https://docs.divergentlabs.xyz/compute/cli/result.md), and nothing a point says changes how a run ends.

### From a Hugging Face Trainer

The Trainer prints `{'eval_loss': 0.0267, 'epoch': 0.04}` to the log, and those numbers stay text there. A callback writes the same dict to the file:

```python
import json, os
from transformers import TrainerCallback

class Telemetry(TrainerCallback):
    def on_log(self, args, state, control, logs=None, **kwargs):
        if state.is_world_process_zero and logs:
            with open(os.environ["DIVERGENT_TELEMETRY_PATH"], "a") as f:
                f.write(json.dumps({"step": state.global_step, **logs}) + "\n")

trainer = Trainer(model=model, args=args, callbacks=[Telemetry()], ...)
```

## Laying it out

A `telemetry.toml` at the top of the directory `div run` ships lays out the Telemetry tab. `--telemetry <file>` names another. It is read when the run is created and kept with it, so a run shows what it was launched with.

```toml
x = "step"                       # what every panel is drawn against

[[panel]]
title = "Loss"
y = ["loss", "eval_loss"]        # up to 4 keys on one panel
scale = "log"

[[panel]]
title = "F1"
y = ["eval_strict_f1", "eval_typed_f1"]
min = 0.9
max = 1.0
rule = 0.95                      # one horizontal line, like a check's threshold

[[panel]]
y = "eval_leak_recall"
wide = true                      # spans both columns

[[stat]]
title = "Samples/s"
y = "eval_samples_per_second"
reduce = "last"                  # last, min or max
```

| Key | In | Means | When left out |
| --- | --- | --- | --- |
| `x` | the file, a panel | the key a panel is drawn against | the point's place in the run |
| `title` | a panel, a stat | what it is called | its keys |
| `y` | a panel, a stat | the key, or keys, to draw | required |
| `scale` | a panel | `linear` or `log` | `linear` |
| `min`, `max` | a panel | the y axis's bounds | fitted to the points |
| `rule` | a panel | a horizontal reference line | none |
| `wide` | a panel | span both columns | `false` |
| `reduce` | a stat | `last`, `min` or `max` | `last` |

A point without a panel's `x` key has nowhere to go on that panel, so it is left off. A key no point has written yet shows as "no points yet", never as a flat line at zero.

A run without a `telemetry.toml` gets one panel per key, in the order the keys first arrived, drawn against `step` when the run writes one. A file that cannot be read refuses the run, the way a check that cannot be read does. That covers a key misspelt, a scale that does not exist, or a `min` above the `max`, and the error says which table and which key:

```text
div: the API refused the run: telemetry [[panel]] 2 has "scael", which is not one of title, x, y, scale, min, max, rule, wide
```

## Reading it

With no keys, `div telemetry` prints every key the run has written, with how many points carry it and the last, lowest and highest value:

```text
key        count  last     min      max
step       412    41200    100      41200
loss       412    0.0412   0.0398   2.31
eval_loss  41     0.0267   0.0261   1.88
```

With keys, it prints the points as tab-separated columns on stdout: `x` first, then one column per key. A point missing a key has an empty cell there, never a zero:

```bash
div telemetry run-7 loss eval_loss > curve.tsv
div telemetry run-7 loss --x epoch
div telemetry run-7 --csv > run-7.csv
```

| Flag | Does |
| --- | --- |
| `--x <key>` | the first column; `step` when the run writes one, the point's place in the run otherwise |
| `--csv` | the same table with commas, quoted where a cell needs it; with no keys, a column for every key the run has written |
| `--json` | every point as one JSON object per line, `{"seq": 1, "data": {...}}` |
| `-f`, `--follow` | keeps printing points as they arrive, until the run ends |

## What is not kept

A line is dropped whole when it is not a JSON object, is over 4 KiB, or has more than 64 numbers. A key that is not letters, digits, `_` and `-` between dots, like `train/loss`, is left out of its point. A run keeps at most 256 keys and 200,000 points.

What is dropped is counted. The log says so the first time each reason comes up, and `div telemetry` repeats the counts on stderr:

```text
telemetry: dropped a line that is not JSON; write one JSON object per line
```

Any value of one of the org's [secrets](https://docs.divergentlabs.xyz/compute/cli/secrets.md) is masked before a line is read, as it is in the log.