Saving Checkpoints
This page covers where to keep model checkpoints during and between training runs, and how to resume from them. The examples use m00xxxx as the project ID; substitute your own.
In short: keep working checkpoints in /scratch/m00xxxx. /scratch is one shared Lustre filesystem, mounted on every login and compute node, so a checkpoint written by one job is visible to the next job on any node. Nothing is tied to the node a run happened on, and nothing needs copying between nodes.
The only node-local storage is the per-job directory /local_scratch/$USER.$SLURM_JOBID, which is deleted when the job ends. That is the one location you must copy checkpoints out of.
Where each kind of checkpoint belongs
| Location | Use it for | Survives the job? | Backed up? |
|---|---|---|---|
/local_scratch/$USER.$SLURM_JOBID |
The fastest I/O during a job, on that node only | No – deleted when the job ends | No |
/scratch/m00xxxx |
Working checkpoints, and resuming between runs | Yes, and visible from every node | No – not replicated, and may be purged |
/projects/m00xxxx |
Checkpoints you want to keep | Yes | Yes, daily |
| Oak, Elm, or cloud storage | Long-term archive of the checkpoints you keep | Yes | Yes |
Don’t write frequent or temporary checkpoints to /projects. Its backups keep copies of deleted files for a while, so checkpoints written and deleted there can keep using your quota after you delete them. Keep that churn on /scratch. If deleted files are still counting against your /projects quota and you need the space back, email srcc-support@stanford.edu or marlowe-info@stanford.edu.
See Filesystems for quotas and details of each filesystem.
The flow
- During the job: write checkpoints to
/scratch/m00xxxx/ckpts/, or write them to/local_scratch/$USER.$SLURM_JOBIDand copy the latest one to/scratch/m00xxxx/ckpts/every few steps. - On start or restart: have your code load the newest checkpoint in
/scratch/m00xxxx/ckpts/. A preempted job onpreemptis put back in the queue and later starts again from the top of its script, so resuming from the latest checkpoint is what makes a preemption cost minutes instead of the whole run. - When the run finishes: copy the checkpoints worth keeping to
/projects/m00xxxxor off the cluster, then delete the rest from/scratch.
Example: a preemptible job script
Adjust the account, GPUs, time, and paths for your own job. Time limits on preempt change with demand; see SLURM on Marlowe for how to check the current limit.
#!/bin/bash
#SBATCH -A marlowe-m00xxxx # your Basic Access account; with a -pmNN account, use -p batch instead
#SBATCH -p preempt # may be preempted and requeued
#SBATCH -G 1
#SBATCH --time=04:00:00
#SBATCH --requeue # requeue if preempted (the default on Marlowe; this makes it explicit)
export CKPT_DIR=/scratch/m00xxxx/ckpts
mkdir -p "$CKPT_DIR"
srun python train.py --ckpt-dir "$CKPT_DIR"
Example: saving and resuming in PyTorch
import glob
import os
import torch
def save_ckpt(step, model, opt, ckpt_dir):
os.makedirs(ckpt_dir, exist_ok=True)
path = os.path.join(ckpt_dir, f"ckpt_{step:08d}.pt")
tmp = path + ".tmp"
torch.save({"step": step, "model": model.state_dict(),
"opt": opt.state_dict()}, tmp)
os.replace(tmp, path) # atomic: a preemption never leaves a half-written checkpoint
def load_latest(model, opt, ckpt_dir):
files = sorted(glob.glob(os.path.join(ckpt_dir, "ckpt_*.pt")))
if not files:
return 0 # no checkpoint yet: start from step 0
ck = torch.load(files[-1], map_location="cpu")
model.load_state_dict(ck["model"])
opt.load_state_dict(ck["opt"])
return ck["step"] + 1 # the step to resume from
Call load_latest once at the start of training and save_ckpt every few steps. Because the step number is zero-padded in the file name, sorting the names finds the newest checkpoint.
Things to watch
/scratchis not backed up and may be purged. Copy anything you need to keep to/projectsor off the cluster.- Moving checkpoints between two projects’
/scratchdirectories:mvis fine for quota, but it keeps the original group. A moved file counts against the new project’s quota, but it keeps its original Unix group, so the new project’s members may not be able to read it. Runchgrp -R marlowe-m00xxxx <dir>with the new project’s ID afterwards, or usecp, thenrm. - Watch the file count, not just space. Thousands of small checkpoint files can reach your inode limit before you run out of space. Delete old checkpoints, and
tarfinished runs into one file. Check both limits with thestorage-quotahelper:
module load marlowe-scripts
storage-quota /scratch/m00xxxx