7  Monitoring, arrays, and interactive jobs

Watching jobs, running many at once, working interactively and on GPUs, and scaling up the right way.

7.1 Watching a job

7.1.1 While it’s queued or running

squeue --me                       # all my jobs, with state and reason
squeue --me -t PENDING            # just the ones still waiting
squeue -j 123456                  # one specific job

The state column: PD pending, R running, CG completing. For pending jobs, the reason in parentheses is what matters:

Reason Meaning What to do
Priority busy machine, you’re in line wait — or pick a better QOS (Section 6.4)
Resources your request is large; waiting for it to free consider right-sizing (Section 5.3)
AssocGrpCpuLimit your account’s core cap is maxed wait for your other jobs to finish
ReqNodeNotAvail requested node is down/reserved drop over-specific constraints
Dependency waiting on another job expected, if you set --dependency

ARCC’s helper turns this into a sentence:

showjobreasons 123456       # "why isn't this running?" in plain language

Follow a running job’s output live:

tail -f slurm-123456.out

The states a job moves through, and how it can end:

stateDiagram-v2
    [*] --> PENDING: sbatch
    PENDING --> RUNNING: your turn
    RUNNING --> COMPLETED: exits 0
    RUNNING --> FAILED: exits non-zero
    RUNNING --> TIMEOUT: hits time limit
    RUNNING --> OUT_OF_MEMORY: exceeds memory
    RUNNING --> PENDING: preempted / requeued
    COMPLETED --> [*]
    FAILED --> [*]
    TIMEOUT --> [*]
    OUT_OF_MEMORY --> [*]
Figure 7.1: A job waits (PENDING), runs, then reaches one terminal state. TIMEOUT and OUT_OF_MEMORY mean you under-requested; preemption sends the job back to PENDING.

7.1.2 After it finishes: did it work, and was it sized right?

sacct -j 123456 --format=JobID,State,Elapsed,MaxRSS,ReqMem,TotalCPU,ExitCode

Key columns: State (COMPLETED, FAILED, TIMEOUT, OUT_OF_MEMORY, CANCELLED), Elapsed (wall time used vs. your --time), MaxRSS (peak memory actually used vs. your --mem), and ExitCode.

seff is a friendlier summary:

seff 123456

It prints CPU and memory efficiency as percentages. Low numbers mean you over-asked — reduce the request next time to start sooner (Section 5.3).

7.1.3 ARCC’s own helpers

These live on your PATH by default and each replaces several squeue/scontrol commands:

showuserjobs                # everything you (and optionally others) have queued/running
showjob 123456              # a rich, readable view of one job
shownode mbcpu-001          # what's running on a given node, and how loaded it is
showuserlimits              # what am I actually allowed to request?
joblist                     # cluster-wide queue overview
pestat                      # per-node CPU/memory load across the cluster

7.1.4 Cancelling

scancel 123456              # cancel one job
scancel -u $USER            # cancel ALL my jobs (careful)
scancel -u $USER --state=PENDING   # cancel only my still-waiting jobs
scancel 123456_7            # cancel one element of a job array (@sec-arrays)

7.1.5 A monitoring routine

For a job you care about:

  1. sbatch submit.sh → note the job ID.
  2. squeue --me → is it PD or R? If PD for a while, showjobreasons <id>.
  3. Running → tail -f slurm-<id>.out.
  4. Done → seff <id> to right-size the next one.

7.2 Job arrays

7.2.1 The problem it solves

You have 500 parameter combinations to sweep, or 500 raster tiles / watersheds / weather-days to process. A bash loop runs them serially in one job, so one crash near the end wastes everything. Copy-pasting 500 submit scripts is no better. An array job is one script, submitted once, that Slurm turns into 500 independent tasks it can run in parallel and backfill wherever there’s room.

7.2.2 The mechanics

Add --array and use $SLURM_ARRAY_TASK_ID to pick which piece of work this task does:

#!/bin/bash
#SBATCH --account=<project>
#SBATCH --job-name=sweep
#SBATCH --array=1-500                 # 500 tasks, indices 1..500
#SBATCH --qos=fast                    # short independent tasks → high-priority QOS
#SBATCH --time=00:30:00               # per task
#SBATCH --cpus-per-task=2
#SBATCH --mem=8G
#SBATCH --output=logs/sweep-%A_%a.out # %A = array job id, %a = task index

module purge
module load gcc/14.2.0 python/3.12.0

# Use the task index to select this task's work:
python run_case.py --case-id ${SLURM_ARRAY_TASK_ID}

Submit once:

mkdir -p logs
sbatch submit_array.sh        # creates 500 tasks: 12345_1, 12345_2, ... 12345_500

Each task is scheduled independently. Because they’re small and short, they backfill well (Section 6.4) and the whole sweep finishes faster than running them in series.

7.2.3 Driving work from the index

The index is just a number; you map it to real work. Common patterns:

Line N of a parameter file:

PARAM=$(sed -n "${SLURM_ARRAY_TASK_ID}p" params.txt)
python run.py $PARAM

The Nth input file:

FILES=(/project/<project>/data/tiles/*.tif)
THIS=${FILES[$SLURM_ARRAY_TASK_ID]}      # note: arrays are 0-indexed in bash
python process.py "$THIS"

(If you index files from a bash array, start --array=0; if you use sed -n Np, start --array=1. Off-by-one here is the classic array bug.)

7.2.4 Throttling

Cap how many tasks run simultaneously with %:

#SBATCH --array=1-500%20        # at most 20 tasks running at a time

This keeps you under your account’s core cap (and protects your fair-share, Section 6.4) while still processing all 500. A modest % limit is often faster overall, because you stop competing with yourself and stay backfill-friendly.

The largest index Slurm allows is 10,000 (MaxArraySize); split bigger sweeps into several arrays.

7.2.5 Managing an array

squeue --me                     # shows pending tasks collapsed, e.g. 12345_[6-500]
scancel 12345                   # cancel the whole array
scancel 12345_7                 # cancel just task 7
sacct -j 12345                  # per-task outcomes after the fact

To find which tasks failed (so you can rerun only those):

sacct -j 12345 --format=JobID,State,ExitCode -n | grep -v COMPLETED

then resubmit just those indices: sbatch --array=17,43,88 submit_array.sh.

Arrays vs. MPI jobs

An array is for independent tasks that don’t communicate: sweeps, ensembles, per-file pipelines. If your tasks must talk to each other — a single domain-decomposed simulation — use an MPI job instead (Section 7.4).

Arrays are also the natural unit for ensembles — fire-weather members, hydrologic scenarios, Monte-Carlo draws — one array task per member, each writing to its own output, collected afterward.

7.3 Interactive sessions and GPU jobs

7.3.1 Interactive sessions with salloc

For hands-on work — debugging a crash, exploring a dataset, building a model interactively — request an interactive allocation:

salloc --account=<project> --qos=interactive \
       --time=02:00:00 --cpus-per-task=4 --mem=16G

salloc requires at least --account and --time. It grants a shell on a compute node, where you can run heavy work off the login node. The interactive QOS starts in ~10–30 seconds (Section 6.2) with an 8-hour ceiling.

Launch salloc from inside tmux

An interactive session lives in your shell. If your laptop sleeps or WiFi drops, the shell dies and the session ends. Start tmux on the login node first, then salloc inside it (Section 2.6). You can then detach, close the laptop, and reattach later with the session intact, as long as it’s within your --time.

When you’re done, exit the shell to release the allocation and free the cores for others.

7.3.1.1 Interactive apps in the portal

The web portal (https://medicinebow.arcc.uwyo.edu) offers graphical interactive apps — a full XFCE desktop, RStudio, and Jupyter — that launch as scheduled jobs behind the scenes. These handle interactive and GPU notebook work without manual salloc and port-forwarding.

7.3.2 GPU jobs

Request GPUs with --gres=gpu:<count> and target a GPU partition:

#!/bin/bash
#SBATCH --account=<project>
#SBATCH --partition=mb-l40s          # or mb-h100, mb-a30, ...
#SBATCH --qos=normal
#SBATCH --gres=gpu:1                 # one GPU on the node
#SBATCH --cpus-per-task=8            # CPUs to feed the GPU
#SBATCH --mem=64G
#SBATCH --time=08:00:00

module purge
module load gcc/13.2.0 cuda-toolkit/12.6.1   # CUDA needs gcc 13.2, see below
nvidia-smi                                    # confirm the GPU is visible
python train.py

To request a specific GPU type you can also use the typed form, e.g. --gres=gpu:h100:2 for two H100s on one node.

7.3.2.1 Choosing a GPU partition

GPU partition GPUs Typical wait Use for
mb-a30 8× A30 minutes development, proof-of-concept, debugging
mb-a6000 / mb-r6000 4× each short–moderate GPU dev, mid-size jobs
mb-l40s 8× L40S ~1 h+ inference, mid-size training
mb-h100 8× H100 seconds–hours large training / production only
Develop on A30s, not H100s

H100 nodes are the most contended resource on the machine — waits of several hours are normal when they’re busy (Section 6.2). Do fast-iteration development on A30s, usually available in minutes, and submit to mb-h100 only for a real training run that needs it. Check gpuavail and showpartitions before choosing.

GPU compiler requirement

cuda-toolkit is built only against gcc/13.2.0, even though gcc/13.2.0 prints a deprecation warning pointing to gcc/14.2.0. For GPU/CUDA work, load gcc/13.2.0 and ignore the warning (Section 4.1). Inside containers, add --nv to expose the GPU (Section 4.2).

7.3.2.2 Checking GPU availability and usage

gpuavail                    # ARCC helper: free GPUs by type, right now
slurm_gpustat               # cluster-wide GPU utilization
sinfo -p mb-h100 -o "%P %t %G %D"   # H100 node states this second
nvidia-smi                  # inside a GPU job: is my code actually using the GPU?
nvtop                       # inside a GPU job: live GPU usage, like htop for GPUs

If nvidia-smi shows 0% utilization while your job runs, your code isn’t actually on the GPU. Verify early.

7.4 The parallelism ladder

7.4.1 Climb only as high as you need

MPI and GPUs sound like “real HPC,” but the right answer is usually lower on the ladder and simpler. Match the rung to the shape of your problem.

flowchart LR
    A["Serial<br/>1 core"] --> B["Job array<br/>many independent runs"]
    B --> C["Multicore<br/>threads, 1 node"]
    C --> D["MPI<br/>1 program, many nodes"]
    D --> E["GPU<br/>ML / PINNs"]
    style B stroke-width:3px,stroke:#4a2c82
Figure 7.2: The parallelism ladder. Climb only as high as the problem needs; most work is solved at the job-array rung (highlighted).

7.4.1.1 Rung 1 — Serial

One core, one task. Fine for small jobs, and the right baseline: get it correct on one core first. If a serial job is fast enough, you’re done — don’t parallelize unnecessarily.

#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1

7.4.1.2 Rung 2 — Job array (many independent serial runs)

If you need to run the same thing many times over different inputs — a parameter sweep, an ensemble, per-file processing — this is almost always the answer. Low complexity, and it backfills well (Section 7.2).

#SBATCH --array=1-500%20
python run.py --id ${SLURM_ARRAY_TASK_ID}

7.4.1.3 Rung 3 — Multicore (one program, many threads, one node)

If a single run is too slow and your code is multithreaded (OpenMP, or numpy/xarray/PyTorch using many cores), give it more cores on one node:

#SBATCH --nodes=1
#SBATCH --cpus-per-task=16
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
./my_threaded_program

Only ask for as many cores as your code actually scales to — check with seff afterward (Section 5.3). Doubling cores rarely doubles speed.

7.4.1.4 Rung 4 — MPI (one program across many nodes)

When a single problem is too big for one node — a domain-decomposed simulation like WRF, where pieces must exchange data every step — you need MPI: one program, many processes across many nodes, communicating.

#SBATCH --nodes=4
#SBATCH --ntasks=384                 # e.g. 96 tasks/node × 4 nodes
#SBATCH --cpus-per-task=1

module purge
module load gcc/14.2.0 openmpi/5.0.10
srun ./my_mpi_simulation             # srun launches the MPI ranks

Use srun (not mpirun) to launch — it’s integrated with Slurm and gets the task layout right automatically. MPI is a step up in complexity; use it only when a problem won’t fit on one node. For long MPI runs, prefer non-investor and checkpoint (Section 8.1).

7.4.1.5 Rung 5 — GPU

For workloads built for GPUs — deep learning, PINNs, some CFD/linear algebra — a GPU can beat dozens of CPU cores, but only if your code actually targets the GPU (Section 7.3.2); a CPU program gains nothing from a GPU node except a longer queue. Develop on mb-a30, scale up to mb-h100 for real training (Section 6.2).

#SBATCH --partition=mb-a30
#SBATCH --gres=gpu:1
module load gcc/13.2.0 cuda-toolkit/12.6.1
python train.py

7.4.2 Choosing a rung

Your situation Rung
It’s already fast enough Serial
“Run this many times over different inputs” Job array (start here)
One run is slow; code uses threads Multicore
One problem too big for a single node; parts must communicate MPI
Code is written for GPUs (ML/PINN) GPU

You can combine rungs — e.g. an array of multicore jobs (a sweep where each run uses 8 cores), which covers a large fraction of real research computing. Start as low as you can; climb only when you need to.