9 Reference

Quick reference: a command cheat sheet, a symptom-to-fix troubleshooting index, and a glossary.

9.1 Cheat sheet

Everything you’ll reach for, on one page.

9.1.1 Connect

ssh <username>@medicinebow.arcc.uwyo.edu     # after SSH Key Manager setup
# Portal: https://medicinebow.arcc.uwyo.edu   (Open OnDemand)
# Keys expire yearly → regenerate in the portal's SSH Key Manager

9.1.2 Storage (nothing is backed up)

arccquota                  # usage vs. limits for home / scratch / project
du -sh * | sort -h         # what's eating my quota here?

Path	Quota	Note
`/home/$USER`	50 GB	small; config only
`/project/<project>`	5 TB	shared group data
`/gscratch/$USER` (`$SCRATCH`)	5 TB	fast, purged after 90 days idle
`/project/<project>/software`	+250 GB	put conda/containers here
`/lscratch`	node-local	wiped at job end

9.1.3 Modules

module spider <name>       # does it exist? what to load first? (use THIS, not avail)
module load gcc/14.2.0 <pkg>
module purge               # clean slate
module save <name> / module restore <name>

9.1.4 Submit and manage jobs

sbatch submit.sh           # submit batch job
salloc -A <project> --qos=interactive -t 2:00:00 -c 4 --mem=16G   # interactive
squeue --me                # my jobs
showjobreasons <id>        # why is it waiting? (plain English)
scancel <id>               # cancel one job (scancel -u $USER = all mine)
seff <id>                  # after the fact: was it sized right?
sacct -j <id> --format=JobID,State,Elapsed,MaxRSS,ReqMem,ExitCode

9.1.5 Minimal submit script

#!/bin/bash
#SBATCH --account=<project>     # REQUIRED — project, not username
#SBATCH --job-name=myjob
#SBATCH --partition=mb               # default CPU pool
#SBATCH --qos=fast                   # 12h, high priority
#SBATCH --time=02:00:00
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G                    # ALWAYS set memory (mb default is 10G/core!)
#SBATCH --output=slurm-%j.out

module purge
module load gcc/14.2.0 python/3.12.0
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
python my_script.py

9.1.6 Choosing partition + QOS

Goal	`--partition`	`--qos`
“Does it run?” (tiny test)	`debug`	`debug` (1 h, instant)
Fast CPU test (≤12 h)	`mb`	`fast`
Interactive dev	`mb`	`interactive`
Normal CPU job (<3 d)	`mb` or `mb,teton`	`normal`
Long CPU sim (3–7 d)	`non-investor`	`long`
GPU dev / proof-of-concept	`mb-a30`	`normal`
GPU production run	`mb-h100` / `mb-l40s`	`normal`/`long`

QOS ceilings & priority: interactive 8h, priority 90 · fast 12h, priority 80 · normal 3d, priority 70 · long 7d, priority 60 · debug 1h, priority 0. Priority is dominated by QOS (weight 200 vs age 10) — pick the shortest tier your job fits. (debug still starts fast because its 4 nodes are dedicated and usually idle, not because of priority.)

9.1.7 GPUs

#SBATCH --partition=mb-l40s
#SBATCH --gres=gpu:1                 # or gpu:h100:2 for a specific type/count
module load gcc/13.2.0 cuda-toolkit/12.6.1   # CUDA needs gcc 13.2 (deprecated but required)
gpuavail                             # free GPUs by type, right now
nvidia-smi / nvtop                   # is my code actually using the GPU?

9.1.8 Job arrays

#SBATCH --array=1-500%20             # 500 tasks, ≤20 at once (max array size 10000)
#SBATCH --output=logs/%A_%a.out
python run.py --id ${SLURM_ARRAY_TASK_ID}

9.1.9 Check the live queue before a big submit

showpartitions             # idle vs busy nodes per partition
gpuavail                   # free GPUs now
sinfo -p mb-h100 -o "%P %t %D"
showuserlimits             # what am I allowed to ask for?

9.1.10 Common mistakes to avoid

--account=<project> is required on every job.
On mb, no --mem → 10 GB per core silently. Always set --mem.
Nothing is backed up. Keep a copy off-cluster.
Install software in /project/<project>/software, not /home (50 GB).
module spider, not module avail.
Long jobs → non-investor partition and checkpoint (investors preempt).
Never compute on the login node. Use sbatch/salloc.
Use tmux so a dropped connection doesn’t kill interactive work.

9.2 Troubleshooting

A symptom-first index to the rest of the book.

9.2.1 Login and access

“Permission denied (publickey)” when I ssh — it worked before. Your SSH key almost certainly expired (they last one year). Regenerate it in the portal’s SSH Key Manager, unzip into ~/.ssh, chmod 700 ~/.ssh && chmod 600 ~/.ssh/*.

“Permission denied” on a brand-new setup. Check permissions: chmod 700 ~/.ssh && chmod 600 ~/.ssh/*, and make sure you unzipped the key into ~/.ssh. Confirm you’re using medicinebow.arcc.uwyo.edu and your SSH key hasn’t expired.

I can’t get an account at all. Account requests for ASCEND/ARID performers are coordinated through the program — ask your project PI, or email Josh Day (josh@rallypoint1.com) or Virginia Iglesias (virginia@innosphere.org). See Section 1.2.

Globus won’t let me add the collection. If you’re an external collaborator on an ARCC-Only account, you can’t use Globus. You need a UW-side guest collection (Section 3.2).

9.2.2 Jobs won’t submit

sbatch: error: ... You have to specify an account. Add --account=<project> (Section 5.2).

Job violates accounting/QOS policy (...) at submit time. Your request exceeds a QOS limit (every QOS is DenyOnLimit). Usually --time exceeds the QOS ceiling — e.g. 5 days under normal (max 3 d). Lower --time or switch QOS (Section 6.1). Check ceilings with sqos and showuserlimits.

Invalid qos specification. Your account can’t use that QOS. Yours are debug, fast, interactive, long, normal (sacctmgr -n show assoc user=$USER format=qos).

9.2.3 Jobs won’t start

Stuck in PENDING with reason Priority. The machine is just busy and you’re in line. Either wait, or pick a higher-priority (shorter) QOS — QOS dominates priority here (Section 6.4). Small/short jobs backfill sooner.

Stuck with reason Resources. You asked for a lot (many cores, much memory, a scarce GPU). Right-size (Section 5.3), or accept the wait. showjobreasons <id> explains in plain language.

GPU job pending for hours. H100/L40S are heavily contended (Section 6.2). Develop on mb-a30 instead, and check gpuavail/showpartitions for what’s actually free before submitting.

Pending with QOSMax... or AssocGrpCpuLimit. You (or your account) hit a cap — e.g. too many cores in flight at once. Wait for your other jobs to finish, or throttle an array with % (Section 7.2).

9.2.4 Jobs fail

OUT_OF_MEMORY. Raise --mem. Compare MaxRSS to ReqMem in sacct (Section 8.2). Watch the 10 GB/core default trap (Section 5.3).

TIMEOUT. Job outran --time. Checkpoint (Section 8.1.3), raise --time within the QOS ceiling, or split the work into an array.

FAILED, worked interactively but not in batch. You loaded modules / activated conda on the login node but not inside the submit script. Put them in the script (Section 5.2, Section 4.2).

Output file references files that “don’t exist.” Relative paths from the wrong directory, or you wrote to /lscratch (wiped at job end). Use absolute paths or $SLURM_SUBMIT_DIR; copy /lscratch output out before the job ends (Section 3.1).

PREEMPTED / REQUEUED on a long job. You were on investor hardware and got reclaimed. Move to --partition=non-investor and checkpoint (Section 8.1).

9.2.5 Software

“That module doesn’t exist” but I’m sure it does. module avail only shows what’s reachable now. Use module spider <name> — you probably need to load a compiler first (Section 4.1).

CUDA won’t build / link. Load gcc/13.2.0 (not 14) — cuda-toolkit is built against 13.2 despite its deprecation warning (Section 4.1).

Home directory full, everything breaking. Conda in /home. Move envs to /project/<project>/software via ~/.condarc, and clean up (Section 4.2). Check with du -sh ~/* | sort -h.

GPU job runs but is slow / nvidia-smi shows 0%. Your code isn’t actually on the GPU. Verify the framework sees CUDA; in a container, you need --nv (Section 7.3.2).

9.2.6 Still stuck?

Email arcc-help@uwyo.edu with your job ID, the submit script, and the error. Or bring it to office hours, Tuesdays 11am–1pm on Zoom. Include what you already tried — it gets you a faster answer.

9.3 Glossary

Plain-language definitions of the terms this book uses.

Account (Slurm): The project that a job is charged to (written <project> in this book’s examples — substitute your own). Required on every job via --account. Not your username. See Section 5.2.
ARCC: Advanced Research Computing Center — the UW group that runs MedicineBow.
ASCEND / ARID: The NSF Regional Innovation Engine (ASCEND) and its wildfire-resilience program (ARID) that many readers of this book work under. See the preface.
Backfill: Slurm slotting a small, short job into a gap in the schedule without delaying higher-priority jobs. Why short jobs start fast. See Section 6.4.
Batch job: A job submitted as a script with sbatch that runs unattended when scheduled. The normal way to run work. See Section 5.2.
Checkpoint: A periodic save of a job’s state so it can resume instead of restart after a preemption, timeout, or crash. Essential for long jobs. See Section 8.1.3.
Compute node: A machine where computation actually runs, allocated to you by Slurm. Distinct from a login node. See Section 1.1.
Condo model: The funding model where research groups buy nodes (inv-*) and others borrow them when idle, subject to reclaim/preemption. See Section 8.1.
Core-hour: One CPU core used for one hour — the unit your allocation is measured in. Reserving 8 cores for 1 hour costs 8 core-hours whether or not you use them.
DenyOnLimit: A QOS flag meaning an over-limit request is rejected at submit time, not silently queued. See Section 6.1.
Fair-share: A priority factor reflecting your account’s recent usage relative to its share; heavy recent use temporarily lowers your priority. See Section 6.4.
Globus: A managed file-transfer service that moves data between “collections” (endpoints), with restart and checksums. Requires a full UW account — ARCC-Only accounts can’t use it. See Section 3.2.
GRES (Generic RESource): How you request GPUs: --gres=gpu:1 or --gres=gpu:h100:2. See Section 7.3.2.
Investor / inv-*: A research group, department, or PI that bought cluster nodes with their own funds. Their inv-* partitions give them top priority; jobs others backfill there can be preempted. See Section 8.1.
Job array: One script run many times with a varying index ($SLURM_ARRAY_TASK_ID), for parameter sweeps and ensembles. Max 10,000 tasks. See Section 7.2.
Lmod / module: The system that loads specific software versions into your shell on demand. Hierarchical: many modules are hidden until you load a compiler. Use module spider. See Section 4.1.
Login node: The shared front-end machine (mblog1, mblog2) you land on when you connect. For editing and submitting — not for computing. See Section 1.1.
MedicineBow: UW/ARCC’s current flagship HPC cluster. The subject of this book.
MPI: Message Passing Interface — the standard for a single parallel program spanning many cores or nodes that communicate. Launched with srun. Contrast with job arrays. See Section 7.4.
OnDemand (Open OnDemand): The web portal at medicinebow.arcc.uwyo.edu — file browser, terminal, and interactive apps.
OOM (Out Of Memory): When a job exceeds its memory reservation and is killed. Fix by raising --mem. See Section 8.2.
Partition: A named pool of nodes with particular hardware (mb, mb-h100, teton, non-investor, …). Chosen with --partition. See Section 6.1.
Preemption: When an investor reclaims a node you had borrowed while it was idle, Slurm stops your running job. On MedicineBow the stopped job is automatically requeued — put back in the queue to start over — rather than simply cancelled. See Section 8.1.
Priority: The score that orders pending jobs. On MedicineBow it’s dominated by QOS (weight 200). See Section 6.4.
QOS (Quality of Service): The service tier that sets your max walltime, resource cap, and priority (debug, fast, normal, long, …). See Section 6.1.
Requeue: Putting a killed (preempted/failed) job back in the pending queue to start over. Pair with checkpointing. See Section 8.1.
Scratch (/gscratch, $SCRATCH): Fast, large, temporary storage purged after 90 days of inactivity. Not backed up. See Section 3.1.
seff: A command showing a finished job’s CPU and memory efficiency — use it to right-size. See Section 7.1.
Slurm: The scheduler that manages all jobs on the cluster. See Section 5.1.
Snapshot: A point-in-time copy of a filesystem, on the same array as the live data. Protects against accidental deletion, not hardware loss. Not a backup. See Section 3.1.
Walltime: Real elapsed clock time a job is allowed (--time). The job is killed at the limit. See Section 5.3.