8  Preemption, checkpointing, and failures

How long jobs survive preemption and walltime limits, and how to diagnose jobs that fail.

8.1 Preemption, the condo model, and checkpointing

8.1.1 The condo model

Much of MedicineBow’s hardware was bought by individual research groups, departments, or PIs using their own grant or startup funds. Those buyers are the investors, and the nodes they paid for make up the inv-* partitions, where they get top priority (priority 110). Everyone else can backfill — run on idle investor nodes that would otherwise sit empty — for free capacity most of the time, with one condition.

An investor can reclaim a node and requeue your job

If you land on investor hardware and the owner submits work, your job is preempted — Slurm kills it and, by default here, requeues it: it returns to the pending queue and starts over from the beginning when resources free up. Any progress not written to disk is lost. This can happen minutes into a job or days in.

8.1.2 Two ways to protect a long run

1. Run where you can’t be preempted. Submit long production jobs to the non-preemptible pool:

#SBATCH --partition=non-investor        # 110 CPU nodes, not reclaimable
# or non-investor-gpu for GPU work

This is the right home for anything long that you haven’t checkpointed. (The --constraint=non-investor some docs mention does not work — it’s a partition, not a constraint. See Section 6.1.)

2. Checkpoint, so preemption is cheap. If you run on borrowable hardware (or list several partitions for faster starts), save state periodically so a requeue is survivable.

8.1.3 Checkpointing

A checkpoint is a periodic save of enough state to resume rather than restart. It protects against preemption, node failures, and hitting your walltime limit. Any job measured in days should checkpoint.

The pattern:

  1. Every so often (every N iterations, or every ~30–60 min of wall time), write current state to a checkpoint file in /gscratch or /project — not /lscratch, which is wiped when the job ends (Section 3.1).
  2. On startup, check for a checkpoint and resume from it if present, otherwise start fresh.
  3. Write checkpoints atomically (write to a temp file, then rename) so a crash mid-write can’t corrupt your only checkpoint. Keep the last one or two.

Many scientific codes have this built in — WRF has restart files, PyTorch has torch.save/load for model+optimizer state, PETSc and Sundials have checkpoint hooks. Turn these on and set the interval to your risk tolerance.

Auto-resubmit on requeue. Combine checkpointing with a self-resubmitting job so a preempted or timed-out job resumes automatically:

#SBATCH --requeue                          # allow this job to be requeued
#SBATCH --open-mode=append                 # don't clobber the log on requeue
#SBATCH --signal=B:USR1@120                # send USR1 120s before the walltime kill

# resume from checkpoint if one exists
if [ -f state.ckpt ]; then
    ./simulate --restart state.ckpt
else
    ./simulate --fresh
fi

The --signal line asks Slurm to warn your script 120 seconds before the walltime kill; trap that signal to flush a final checkpoint.

8.1.4 Requeue behavior worth knowing

  • A requeued job keeps its job ID but starts its command from the beginning — Slurm doesn’t magically resume your program; your checkpoint does.
  • --open-mode=append stops the requeued run from overwriting the previous log.
  • Check whether a job was requeued with sacct -j <id> --format=JobID,State,Start,End — you’ll see multiple rows.
Checkpoint jobs that run longer than a few hours

Short jobs (minutes to an hour or two) can just be rerun if something goes wrong. Any multi-hour or multi-day job should checkpoint, and any long production run should also prefer non-investor. A non-preemptible partition plus checkpoints is what lets big simulations finish reliably on a shared machine.

8.2 When jobs fail

8.2.1 Step one: what state did it end in?

sacct -j 123456 --format=JobID,State,ExitCode,Elapsed,MaxRSS,ReqMem,DerivedExitCode

The State column is your first clue:

State Meaning Usual fix
COMPLETED exit code 0 none (but check the output is correct)
OUT_OF_MEMORY killed by the OOM killer raise --mem (see below)
TIMEOUT hit the --time limit raise --time, or checkpoint (Section 8.1.3)
FAILED your program exited non-zero read the .out/.err file — it’s your bug
NODE_FAIL the hardware died resubmit; checkpoint so it’s cheap
PREEMPTED / REQUEUED investor reclaimed the node Section 8.1
CANCELLED someone ran scancel (maybe you) —

8.2.2 Out of memory

A job that exceeds its memory reservation is killed by the Linux OOM killer — you’ll see OUT_OF_MEMORY in sacct, and often a truncated or empty output file.

sacct -j 123456 --format=JobID,State,MaxRSS,ReqMem
# MaxRSS is peak memory actually used; compare to ReqMem

If MaxRSS is near or above ReqMem, raise --mem and resubmit. Then use seff (Section 7.1) on the successful run to right-size going forward. Note the 10 GB/core default on mb (Section 5.3) — sometimes you already had enough memory and something else was wrong.

8.2.3 Timeout

TIMEOUT means your job was still running when the clock ran out and Slurm killed it. Options, in order of preference:

  1. Checkpoint so the next run resumes instead of restarting (Section 8.1.3). This is the durable fix for genuinely long work.
  2. Raise --time — but staying under your QOS ceiling (normal = 3 d, long = 7 d), and knowing a longer request may wait longer to start (Section 6.2).
  3. Make the job finish faster — more cores if it scales, or a smaller problem per job (an array, Section 7.2).

8.2.4 Your program crashed (FAILED)

FAILED with a non-zero exit code is usually your code, not the cluster. Read the output:

cat slurm-123456.out          # stdout+stderr (or separate --error file)

Common cluster-specific culprits:

  • Module not loaded in the script. Interactive worked, batch didn’t — because you loaded modules on the login node but not inside the submit script. Put module load in the script (Section 5.2).
  • Wrong paths. A relative path that existed where you ran sbatch but not where the job ran. Use absolute paths or $SLURM_SUBMIT_DIR.
  • conda env not activated in the script (Section 4.2).
  • Wrote to /lscratch and it vanished — that’s node-local and wiped at job end (Section 3.1).

8.2.5 Node failure and preemption

NODE_FAIL (hardware died) and PREEMPTED/REQUEUED (investor reclaimed the node, Section 8.1) are not your bug. The response is the same: checkpoint so a restart is cheap, run long jobs on non-investor, and add --requeue so they come back automatically. If node failures recur on the same node, report it to arcc-help@uwyo.edu with the job ID.

8.2.6 A debugging loop

  1. sacct -j <id> ... → get the State, narrowing it to one of the four families above.
  2. Resource-related (OOM/timeout) → adjust --mem/--time, resubmit.
  3. FAILED → read slurm-<id>.out, fix the code/env, and reproduce it fast on the debug partition (near-instant start, Section 6.2) rather than waiting in a long queue each iteration.
  4. Once it works, seff <id> → right-size → promote to the real partition/QOS.

Use the debug partition as your fast edit-run-debug loop, and graduate to big queues only once the job is known-good.