# Chart training metrics

> Record loss, learning rate and other metrics during a run, lay out the run's charts, and export the values.

This guide shows how to record metrics while a run trains, so they appear as charts on the run's **Telemetry** tab in the dashboard and in `div telemetry`.

## 1. Append metrics to the telemetry file

Every run has the environment variable `DIVERGENT_TELEMETRY_PATH`. Append one JSON object per line:

```python
import json, os

def record(**values):
    with open(os.environ["DIVERGENT_TELEMETRY_PATH"], "a") as f:
        print(json.dumps(values), file=f, flush=True)

record(step=120, loss=0.41, lr=3e-4)
```

Each line becomes one point. Nested objects become dotted keys: `{"eval": {"loss": 0.4}}` is `eval.loss`. Non-numeric values are ignored. Points appear about one second after they are written.

Numbers printed to the log are not charted. Only the telemetry file is read.

### With a Hugging Face Trainer

Add a callback that writes each log entry to the file:

```python
import json, os
from transformers import TrainerCallback

class Telemetry(TrainerCallback):
    def on_log(self, args, state, control, logs=None, **kwargs):
        if state.is_world_process_zero and logs:
            with open(os.environ["DIVERGENT_TELEMETRY_PATH"], "a") as f:
                f.write(json.dumps({"step": state.global_step, **logs}) + "\n")

trainer = Trainer(model=model, args=args, callbacks=[Telemetry()], ...)
```

## 2. Lay out the charts (optional)

Without a layout, the Telemetry tab shows one chart per key, plotted against `step` when the run records it. To choose the charts, add a `telemetry.toml` to the root of the directory you run from:

```toml
x = "step"

[[panel]]
title = "Loss"
y = ["loss", "eval_loss"]
scale = "log"

[[stat]]
title = "Samples/s"
y = "eval_samples_per_second"
```

The layout is saved with the run when it starts. If the file is invalid, the run is rejected and the error names the table and key to fix. See [telemetry.toml](https://docs.divergentlabs.xyz/compute/cli/telemetry.md#telemetrytoml) for every key.

## 3. Read the values

```bash
div telemetry run-7                        # every key, with count, last, min and max
div telemetry run-7 loss eval_loss         # tab-separated columns
div telemetry run-7 loss -f | tail -1      # follow while the run trains
div telemetry run-7 --csv > run-7.csv      # every key, as CSV
```

Telemetry is for charts only. It does not affect the run's status or its [checks](https://docs.divergentlabs.xyz/compute/guides/results.md).

## Related

- [`div telemetry`](https://docs.divergentlabs.xyz/compute/cli/telemetry.md), including the limits on lines, keys and points