Stanford University · Stanford Institute for Human-Centered AIStanford HAI

Consolidating Data to Reduce Inode Usage

Storing a dataset as many tiny files – one file per array chunk, per training sample, or per checkpoint step – uses up your inode (file-count) quota, often long before you run out of space. It also slows every read, because the filesystem has to open thousands of separate files. This page shows how to store the same data in fewer, larger files.

To check how close you are to the file-count limit, run storage-quota (see the FAQ).

The Python examples below use numpy, zarr, h5py, and webdataset. Install whichever you need into your own environment, for example:

python3 -m venv ~/venvs/repack && source ~/venvs/repack/bin/activate
pip install numpy zarr h5py webdataset

General principle: fewer, larger files

Pack data into a modest number of large files. A good rule of thumb is shards of roughly 100 MB–1 GB each, read sequentially. The WebDataset layout is a common way to do this: group each sample’s parts into a record and write the records into sharded .tar files.

import webdataset as wds

# Write ~1 GB shards: shards-000000.tar, shards-000001.tar, ...
with wds.ShardWriter("shards-%06d.tar", maxsize=1e9) as sink:
    for key, image_bytes, label in samples:
        sink.write({"__key__": key, "jpg": image_bytes, "cls": label})

For a directory you are finished with, a plain tar is enough:

tar -cf dataset.tar dataset/ && rm -r dataset/

Zarr

A Zarr array stored as a directory writes one file per chunk, so small chunks produce huge numbers of files. There are three fixes, often combined:

  • Bigger chunks: fewer chunk files, at the cost of coarser random access.
  • A single-file store: ZipStore packs the whole array into one .zip.
  • Consolidated metadata: for a group that holds many arrays, gather every array’s metadata into one file at the top of the group (.zmetadata in Zarr format 2, zarr.json in format 3).
import zarr

# Bigger chunks -> fewer chunk files
zarr.open("data.zarr", mode="w", shape=(1_000_000, 512),
          chunks=(16_384, 512), dtype="f4")

# Or store the whole array as a single file
store = zarr.storage.ZipStore("data.zip", mode="w")
zarr.open(store=store, mode="w", shape=(1_000_000, 512),
          chunks=(16_384, 512), dtype="f4")
store.close()

# For a group of many arrays, consolidate the metadata, then open it that way
zarr.consolidate_metadata("group.zarr")
group = zarr.open_consolidated("group.zarr")

To rechunk an existing large store without loading it all into memory, see the rechunker package.

HDF5 (.h5)

HDF5 is already a single-file container, so the usual problem is having many small .h5 files, one per sample or step. Merge them into one file, with chunking and compression enabled:

import glob
import h5py

with h5py.File("dataset.h5", "w") as out:
    for i, path in enumerate(sorted(glob.glob("samples/*.h5"))):
        with h5py.File(path, "r") as f:
            out.create_dataset(f"sample_{i}", data=f["data"][:],
                               chunks=True, compression="gzip")

To re-chunk or compress an existing file, use h5repack from the hdf5_18 module:

module load hdf5_18
h5repack -f GZIP=4 in.h5 out.h5

NumPy (.npy / .npz)

One .npy file per sample is the classic inode blow-up. If the arrays share a shape and dtype, write them into a single .npy file one sample at a time. open_memmap creates the full-size file on disk and fills it in place, so memory use stays flat however large the dataset is:

import glob
import numpy as np

files = sorted(glob.glob("samples/*.npy"))
first = np.load(files[0])

# Create the file on disk, then fill it one sample at a time
out = np.lib.format.open_memmap("dataset.npy", mode="w+", dtype=first.dtype,
                                shape=(len(files), *first.shape))
for i, f in enumerate(files):
    out[i] = np.load(f)
out.flush()
del out

big = np.load("dataset.npy", mmap_mode="r")  # slice without loading everything

A memory-mapped single file is usually fastest for random-access training. For a small set of arrays with different shapes, a single .npz archive also works, but np.savez holds every array in memory while it writes, and each array is read in full when you access it:

np.savez("dataset.npz", **{f"s{i}": np.load(f) for i, f in enumerate(files)})

Where to do the repack

Repacking is metadata-heavy, because it opens every one of those small files. Do it inside a job on the node-local scratch directory rather than directly on Lustre, then copy the single consolidated result back:

WORK="/local_scratch/$USER.$SLURM_JOBID"   # per job, wiped when the job ends
cp big-dataset.tar "$WORK"/ && cd "$WORK" && tar -xf big-dataset.tar
# ... repack into one consolidated file (see above) ...
cp dataset.zip /scratch/<project>/         # copy the result back

Once you have checked that the consolidated copy is complete (for example, by opening it and comparing the sample count), delete the original small files. Removing them is what frees the inodes:

rm -r /scratch/<project>/samples/          # the original small files