The runtime

Image

pytorch/pytorch:2.14.0-cuda12.6-cudnn9-runtime: Python, PyTorch 2.14, CUDA 12.6 and cuDNN 9, on every SKU.

--image replaces it with any public image that has sh, and python for a .py entrypoint. Other dependencies are installed by the image or by the command:

div run -- 'pip install -r requirements.txt && python train.py'

The image’s Python is Ubuntu’s, which marks itself as externally managed, so pip would refuse to install into it. Every run is started with PIP_BREAK_SYSTEM_PACKAGES=1, which lets it: the container is thrown away when the run ends, so there is no system to break.

Hardware

SKU	GPU	CPU	Memory
b300, h200, h100	--gpus of them	1 core per GPU	8 GiB per GPU
cpu	—	1 core	2 GiB

That is the container a run gets when it does not ask. --cpu and --memory ask for another, up to 64 cores and 336 GiB, at $0.17 an hour for each core past these and $0.03 for each GiB: see The container. A run is held to its container: past its memory it is killed, and past its cores it is slowed.

Filesystem

Path	What it holds	Kept
/workspace	the shipped code; the command starts here	no
/mnt/<volume>	every volume the org has	yes
/tmp/divergent/result.json	the run’s result	as the result
/tmp/divergent/telemetry.jsonl	the run’s telemetry, one JSON object per line	as points

With --no-upload, the command starts in the image’s working directory.

Volumes are mounted when the run starts, so /mnt/<volume> is a volume only if one by that name exists by then. Any other directory under /mnt is the container’s own disk, and what is written there goes when the run ends.

A volume the command names is made before the run is queued, the way div storage put makes the one it names, and the log says so:

div run -- 'mkdir -p /mnt/mask-models/base && python download.py'
new volume mask-models, for the /mnt/mask-models the command names

A path the code builds for itself is not read. Make that volume first, with div storage put mask-models <path> or a command that names it. If a run leaves anything in a directory under /mnt that is not a volume, the last lines of its log name the directory and how much was lost:

/mnt/mask-models is not a volume, so the 1.1G the run wrote there went with the container. Volumes are mounted when a run starts: make mask-models first with div storage put mask-models <path>, or name /mnt/mask-models in the command.

Environment

Later rows win:

Source	Variables
The runtime	PIP_BREAK_SYSTEM_PACKAGES=1
Secrets	every secret the org has
--env	each KEY=value given
The run	DIVERGENT_RUN_ID, DIVERGENT_RUN_NAME, DIVERGENT_RESULT_PATH, DIVERGENT_TELEMETRY_PATH
An eval run	DIVERGENT_MODEL, DIVERGENT_MODEL_VERSION_ID, DIVERGENT_MODEL_PATH, DIVERGENT_MODEL_DIGEST, DIVERGENT_EVALUATION

Lifetime

The command runs under sh -lc. A run ends when it exits, is stopped, reaches its --max-spend, or has run for 24 hours. Its exit code is div run’s.