Stanford University · Stanford Institute for Human-Centered AIStanford HAI

Saving Checkpoints

This page covers where to keep model checkpoints during and between training runs, and how to resume from them. The examples use m00xxxx as the project ID; substitute your own.

In short: keep working checkpoints in /scratch/m00xxxx. /scratch is one shared Lustre filesystem, mounted on every login and compute node, so a checkpoint written by one job is visible to the next job on any node. Nothing is tied to the node a run happened on, and nothing needs copying between nodes.

The only node-local storage is the per-job directory /local_scratch/$USER.$SLURM_JOBID, which is deleted when the job ends. That is the one location you must copy checkpoints out of.

Where each kind of checkpoint belongs

Location Use it for Survives the job? Backed up?
/local_scratch/$USER.$SLURM_JOBID The fastest I/O during a job, on that node only No – deleted when the job ends No
/scratch/m00xxxx Working checkpoints, and resuming between runs Yes, and visible from every node No – not replicated, and may be purged
/projects/m00xxxx Checkpoints you want to keep Yes Yes, daily
Oak, Elm, or cloud storage Long-term archive of the checkpoints you keep Yes Yes

Don’t write frequent or temporary checkpoints to /projects. Its backups keep copies of deleted files for a while, so checkpoints written and deleted there can keep using your quota after you delete them. Keep that churn on /scratch. If deleted files are still counting against your /projects quota and you need the space back, email srcc-support@stanford.edu or marlowe-info@stanford.edu.

See Filesystems for quotas and details of each filesystem.

The flow

  1. During the job: write checkpoints to /scratch/m00xxxx/ckpts/, or write them to /local_scratch/$USER.$SLURM_JOBID and copy the latest one to /scratch/m00xxxx/ckpts/ every few steps.
  2. On start or restart: have your code load the newest checkpoint in /scratch/m00xxxx/ckpts/. A preempted job on preempt is put back in the queue and later starts again from the top of its script, so resuming from the latest checkpoint is what makes a preemption cost minutes instead of the whole run.
  3. When the run finishes: copy the checkpoints worth keeping to /projects/m00xxxx or off the cluster, then delete the rest from /scratch.

Example: a preemptible job script

Adjust the account, GPUs, time, and paths for your own job. Time limits on preempt change with demand; see SLURM on Marlowe for how to check the current limit.

#!/bin/bash
#SBATCH -A marlowe-m00xxxx      # your Basic Access account; with a -pmNN account, use -p batch instead
#SBATCH -p preempt              # may be preempted and requeued
#SBATCH -G 1
#SBATCH --time=04:00:00
#SBATCH --requeue               # requeue if preempted (the default on Marlowe; this makes it explicit)

export CKPT_DIR=/scratch/m00xxxx/ckpts
mkdir -p "$CKPT_DIR"
srun python train.py --ckpt-dir "$CKPT_DIR"

Example: saving and resuming in PyTorch

import glob
import os

import torch

def save_ckpt(step, model, opt, ckpt_dir):
    os.makedirs(ckpt_dir, exist_ok=True)
    path = os.path.join(ckpt_dir, f"ckpt_{step:08d}.pt")
    tmp = path + ".tmp"
    torch.save({"step": step, "model": model.state_dict(),
                "opt": opt.state_dict()}, tmp)
    os.replace(tmp, path)  # atomic: a preemption never leaves a half-written checkpoint

def load_latest(model, opt, ckpt_dir):
    files = sorted(glob.glob(os.path.join(ckpt_dir, "ckpt_*.pt")))
    if not files:
        return 0  # no checkpoint yet: start from step 0
    ck = torch.load(files[-1], map_location="cpu")
    model.load_state_dict(ck["model"])
    opt.load_state_dict(ck["opt"])
    return ck["step"] + 1  # the step to resume from

Call load_latest once at the start of training and save_ckpt every few steps. Because the step number is zero-padded in the file name, sorting the names finds the newest checkpoint.

Things to watch

  • /scratch is not backed up and may be purged. Copy anything you need to keep to /projects or off the cluster.
  • Moving checkpoints between two projects’ /scratch directories: mv is fine for quota, but it keeps the original group. A moved file counts against the new project’s quota, but it keeps its original Unix group, so the new project’s members may not be able to read it. Run chgrp -R marlowe-m00xxxx <dir> with the new project’s ID afterwards, or use cp, then rm.
  • Watch the file count, not just space. Thousands of small checkpoint files can reach your inode limit before you run out of space. Delete old checkpoints, and tar finished runs into one file. Check both limits with the storage-quota helper:
module load marlowe-scripts
storage-quota /scratch/m00xxxx