# div run

> Places a run, ships the working directory to it, and stays attached to its log until it stops. Exits with the run's own exit code.

```bash
div run --gpu h100 train.py::run --max-spend 5.00
div run --gpu h200 --gpus 32 finetune.py::lora --max-spend 1500
div run --cpu 8 --memory 64G train.py --max-spend 10
div run --detach -- torchrun --nproc-per-node 8 train.py
div run eval.py --check 'metrics.accuracy>=0.8'
```

`div run` places a run and stays attached to its log until it stops, then exits with the run's own exit code — a run that exits 3 makes `div run` exit 3.

Everything that can be wrong with what was typed is found before anything is created or uploaded. A misspelled flag is refused rather than ignored: `--max-spent 5` should not place a run with no ceiling.

## Flags

| Flag | What it does |
| --- | --- |
| `--gpu <sku>` | `b300`, `h200`, `h100` or `cpu` (default `h100`) |
| `--gpus <n>` | how many of them (default 1) |
| `--cpu <n>` | physical cores for the container (default: what comes with the SKU). See [The container](#the-container) |
| `--memory <GiB>` | memory for the container in whole GiB, like `64G` (default: what comes with the SKU) |
| `--max-spend <usd>` | stop the run when it has cost this much |
| `--project <slug>` | the project it belongs to (default: this directory's name) |
| `--name <name>` | what to call it (default: one the control plane picks) |
| `--image <ref>` | the container image to run in (default `pytorch/pytorch:2.14.0-cuda12.6-cudnn9-runtime`). See [The runtime](https://docs.divergentlabs.xyz/compute/runtime.md) |
| `--env KEY=value` | an environment variable for the run; repeatable. It is kept with the run and shown back, so a credential belongs in a [secret](https://docs.divergentlabs.xyz/compute/cli/secrets.md), which every run has too. `--env` wins over a secret of the same name |
| `--check <check>` | what the run's result must say, like `metrics.accuracy>=0.8`; repeatable. See [Checks](#checks) |
| `--telemetry <file>` | how the run's Telemetry tab is laid out (default: the `telemetry.toml` shipped, if there is one). See [div telemetry](https://docs.divergentlabs.xyz/compute/cli/telemetry.md#laying-it-out) |
| `--volume <name>` | the volume the code is shipped to (default `<project>-code`) |
| `--no-upload` | run the image as it is, ship nothing |
| `-d`, `--detach` | print the run's name and return |
| `--json` | print the run as JSON |

`--gpu h100` and `--gpu=h100` are the same flag. Everything after `--` is the run's, not ours, so a flag meant for your own command never reaches `div`.

## The hardware

| SKU | Per hour |
| --- | --- |
| `b300` | $8.95 |
| `h200` | $5.95 |
| `h100` | $4.95 |
| `cpu` | $0.30 |

That is per GPU, metered by the minute. `--gpus 8 --gpu h100` costs $39.60 an hour. Runs are placed in GPUs, not nodes: `--gpus 32` asks for the same hardware as four nodes of eight, and `--nodes` is refused with that sentence.

An unrecognized SKU is refused, along with the list of the ones the control plane accepts.

## The container

Each GPU comes with one core and 8 GiB, and the `cpu` SKU with one core and 2 GiB, so `--gpus 4` gets four cores and 32 GiB. That is in the price above. `--cpu` and `--memory` ask for a different container, and each core past what comes with the SKU is $0.17 an hour, each GiB $0.03:

```text
div run --gpu h100 --cpu 8 --memory 64G train.py
run-k4t9  H100 · 8 cores · 64 GiB  $7.82/h  no ceiling
```

That is $4.95 for the H100, 7 more cores at $0.17 and 56 more GiB at $0.03. Asking for less than comes with the SKU costs the same as asking for nothing.

The container is held to what it asked for. A run that needs more memory than it has is killed, and says `killed (out of memory, or stopped)`; one that wants more cores is slowed to the ones it has. A GPU spent waiting on one core for its next batch is the most expensive way to tokenize, so a run with a heavy dataloader asks for the cores up front. Cores are physical, two threads each. The most one container takes is 64 cores and 336 GiB; `--memory` takes `64`, `64G` or `64GiB`, and `512M` is refused rather than rounded up.

## The ceiling is dollars

`--max-spend 5.00` is five dollars, and the control plane stops the run there. `5`, `5.00` and `$5` are the same ceiling. It becomes 500 cents at the command line by reading the digits rather than multiplying a float, and nothing downstream sees a fraction; a third decimal rounds half up.

Before the first line of output arrives, `div` prints what the run costs an hour and how long the ceiling buys at that rate:

```text
run-k4t9  H100  $4.95/h  $5.00 buys 1h 0m
```

A run with no `--max-spend` says `no ceiling` there, and runs until it exits, someone stops it, or it has run for 24 hours — the longest any run is given.

## The directory is the project

Runs started in `~/very-cool-classifier` belong to the project `very-cool-classifier`, which is created the first time you run there. The name is slugified the same way the API does it, so what you type and what the control plane stores are the same string.

`--project`, `DIVERGENT_PROJECT`, or a `divergent.json` in this directory or one above it overrides the directory name. See [Configuration](https://docs.divergentlabs.xyz/compute/cli/configuration.md).

## The working directory is shipped

Everything `git ls-files` would list — tracked files, plus untracked ones git is not ignoring — is uploaded to a volume before the run is placed, into its own directory per run. The run starts in a copy of it at `/workspace`:

```text
14 files, 38.2 KB → very-cool-classifier-code/code/9f31c2ab
```

Every run of a project gets its own directory on one volume, so two runs started a second apart do not read each other's code. The volume is the ordinary kind: `div storage get` can read back what a run was given.

`/workspace` belongs to the container and goes when it does. A checkpoint written to `./ckpt` is lost with it; one written to `/mnt/<volume>/ckpt` is kept. [The runtime](https://docs.divergentlabs.xyz/compute/runtime.md) has the rest of what a run starts with.

Outside a repository the walk skips `.git`, `node_modules`, `__pycache__`, `.venv`, `venv`, tool caches, `dist`, `.next`, `.DS_Store`, `*.pyc` and anything beginning with `.env`. Everything else goes, data included: a 6 GB `data/` beside `train.py` is 6 GB uploaded on every run. `git init` and a `.gitignore` that names it keeps it here; [`div storage put`](https://docs.divergentlabs.xyz/compute/cli/storage.md) sends it to a volume once, and every run reads it from `/mnt/<volume>`. The count printed before the upload is the thing to check.

A directory with nothing to send is refused — run it where the code is, or pass `--no-upload` to run the image as it is.

## The entrypoint is read three ways

| Written | What the container is told |
| --- | --- |
| `train.py::run` | `python -u -c 'from train import run; run()'` |
| `train.py` | `python -u train.py` |
| `torchrun --nproc-per-node 8 train.py` | your own command, passed through to the shell unchanged |

`file.py::function` imports the module and calls the function; a path like `models/lora.py::train` imports `models.lora`. Anything with a space in it is your own command. Python runs unbuffered either way, because a log that arrives in one lump at the end is not a log worth following.

The control plane keeps what you typed next to what the container was told. A run's page in the dashboard shows `train.py::run` rather than the `cd` into that run's copy of the code, and offers the `div run` that places it again from where the code is now.

## Attached, detached, and Ctrl-C

Attached, `div` prints the run's log as it arrives and ends with one line about how it went:

| Status | The last line |
| --- | --- |
| succeeded | `run-k4t9 finished in 12m 40s and spent $1.05`, then `· missed 1 of 2 checks` when it had checks |
| hit its ceiling | `run-k4t9 hit its $5.00 ceiling after 1h 0m` |
| stopped by the org | `run-k4t9 was stopped by the org after 12m 0s: org usage limit of $500.00 reached — $3.10 spent` |
| failed | `run-k4t9 failed after 3m 12s (exit 1): … — $0.27 spent` |
| canceled | `run-k4t9 was canceled after 4m 0s, $0.33 spent` |

`--detach` prints the run's name on stdout and returns once it is placed. Ctrl-C detaches too, leaving the run going to its ceiling, and on the way out prints the commands to watch it again and to stop it. [`div stop`](https://docs.divergentlabs.xyz/compute/cli/stop.md) stops it; [`div runs`](https://docs.divergentlabs.xyz/compute/cli/runs.md) finds it again if you lost the name.

## Checks

A run that exits 0 has shown it did not crash, which is not the same as showing the experiment worked. `--check` says what its result must say for that:

```bash
div run eval.py --check 'metrics.accuracy>=0.8' --check 'metrics.loss<0.6'
```

Every run is given `DIVERGENT_RESULT_PATH`, and a JSON object written there is its [result](https://docs.divergentlabs.xyz/compute/cli/result.md). Once the run exits, the control plane judges each check against it — not the run, which does not grade itself — and writes the verdict into the log:

```text
check passed: metrics.accuracy >= 0.8 (found 0.91)
check failed: metrics.loss < 0.6 (found 0.71)
```

A check is a path into the result, one comparison — `>=`, `<=`, `>`, `<`, `==` or `!=` — and a number, `true`, `false`, or a string. Two conditions are two checks, and every one must pass. A path the result does not have is unknown, and unknown is never a pass: a run that forgot to write its accuracy has not shown it is accurate.

A check is the experiment's verdict, never the run's. A run that exits 0 and misses one is a good run of an experiment that did not pass: it stays `succeeded`, `div run` still exits 0, and the last line says `missed 1 of 2 checks`. The verdicts are shown where experiments are read — the run's result, and for an eval run its [evaluation](https://docs.divergentlabs.xyz/compute/cli/eval.md). A run that exits non-zero is `failed` with its own exit code, and its checks are still judged, because a failed run's numbers are worth reading.

## JSON

`--json` prints the run as it ended, as the API sent it. With `--detach` it prints the run as it was placed and returns. The CLI keeps every field it does not name, so a field the API adds shows up here without a new build of `div`.