divergent/docs
ComputePrivate beta

Keep data and checkpoints on a volume

Upload a dataset once, read it from every run, save checkpoints that outlast the run, and download them.

This guide shows how to use a volume to keep files between runs. A volume is persistent storage that belongs to your org. Every run can read and write every volume at /mnt/<volume>.

Files a run writes anywhere else, including its working directory /workspace, are deleted when the run ends.

Upload a dataset once

div run uploads your working directory on every run. Keep large data out of it and upload the data to a volume instead:

div storage put training-data ./data

div storage put creates the volume if it does not exist. The directory lands under its own name, so the files are at /mnt/training-data/data/ in a run.

If your code is in a git repository, add the data directory to .gitignore so div run does not upload it. Outside a repository, move the data out of the directory you run from.

Read it from a run

import pandas as pd

df = pd.read_parquet("/mnt/training-data/data/train.parquet")

Save checkpoints

Write checkpoints under /mnt/<volume>:

trainer.save_model("/mnt/checkpoints/lora-v1/final")

The volume must exist when the run starts. A run cannot create one while it is running. To make sure it exists, do one of these:

  • Create it first with div storage put checkpoints <path>.

  • Name the path in the run’s command. div run creates any volume that appears as /mnt/<volume> in the command before the run starts:

    div run -- 'mkdir -p /mnt/checkpoints/lora-v1 && python train.py'

If a run writes to a directory under /mnt that is not a volume, the files are lost when the run ends, and the end of the run’s log says how much was lost.

Download the results

div storage ls checkpoints lora-v1
div storage get checkpoints lora-v1/final

div storage get downloads a directory into a local directory of the same name. Downloads have no size limit.

Large files

Each uploaded file can be at most 256 MB, and each file must finish uploading within 5 minutes. For larger datasets or base weights, download them inside a run and write them straight to a volume:

div run --gpu cpu --no-upload -- 'pip install huggingface_hub && huggingface-cli download Qwen/Qwen2.5-7B --local-dir /mnt/weights/qwen2.5-7b'
esc