4  Software: modules, conda, and containers

How software is provided on the cluster, and how to install your own without filling your home directory.

4.1 Software modules

4.1.1 Why modules exist

A cluster has hundreds of users who need incompatible software: three versions of Python, five compilers, conflicting builds of GDAL. Modules let you load exactly the versions you want into your shell for this session, without affecting anyone else. Load a module and its programs appear on your PATH; unload it and they vanish.

module load gcc/14.2.0        # add gcc 14 to this shell
module list                   # what do I currently have loaded?
module unload gcc/14.2.0      # remove it
module purge                  # clear everything and start clean

4.1.2 The hierarchy

MedicineBow uses Lmod in hierarchical mode. Modules are arranged in layers:

  • Core modules are always visible: the compilers (gcc, nvhpc-sdk, oneapi), MPI, and standalone tools.
  • Compiler-dependent modules — Python, R, GDAL, NetCDF, HDF5, CUDA — are hidden until you load a compiler. They were built against a specific compiler, so Lmod won’t show them until you’ve picked one.

So a new user runs module avail, doesn’t see gdal, and concludes it isn’t installed. It is — just one layer down.

Use module spider, not module avail, to check whether something exists

module avail shows only what’s loadable given what you’ve already loaded. module spider searches the entire tree and tells you both that something exists and what to load first to reach it.

module spider gdal            # yes it exists; here are the versions
module spider gdal/3.13.0     # ...and exactly what to load before it

A typical geospatial stack, in order:

module purge
module load gcc/14.2.0 openmpi/5.0.5   # 1. compiler + MPI first
module load gdal/3.7.3 netcdf-c/4.9.2-ompi   # 2. now these are reachable

4.1.3 What’s installed (a tour)

Ask the machine for the current list with module spider, but roughly:

  • Compilers (core): gcc/{11.4.0, 11.5.0, 13.2.0, 14.2.0, 15.2.0}, nvhpc-sdk, oneapi, openmpi/5.0.10.
  • After loading a compiler: python/{3.10.6, 3.12.0, 3.14.4}, r/{4.4.0, 4.5.3}, gdal/{3.7.3, 3.13.0}, netcdf-c/4.9.2-ompi (also needs an openmpi module), hdf5/1.14.3*, cuda-toolkit/{11.7.1, 12.4.1, 12.6.1, 12.8.0}, openmpi/4.1.6, rclone/1.63.1.
  • Standalone: apptainer/{1.3.5, 1.4.1}, singularity/4.1.0, miniconda3/24.3.0, julia/{1.10.3, 1.11.6}.
CUDA needs the deprecated gcc/13.2.0

gcc/13.2.0 prints a deprecation warning pointing to gcc/14.2.0, but CUDA 12.4.1 and 12.6.1 are built only against gcc/13.2.0 — for those versions you must load the deprecated compiler:

module load gcc/13.2.0        # the one that warns you — CUDA 12.4/12.6 need it
module load cuda-toolkit/12.6.1

Ignore the warning here; it’s a packaging quirk. (The newest, cuda-toolkit/12.8.0, is built against gcc/14.2.0, so if you can use 12.8.0 you avoid the deprecated compiler.)

4.1.4 Making a set of modules load automatically

Once you know the stack a project needs, save it as a named collection so you don’t retype it:

module purge
module load gcc/14.2.0 openmpi/5.0.5 gdal/3.7.3 netcdf-c/4.9.2-ompi
module save geostack          # remember this set as "geostack"

# next session:
module restore geostack

Prefer module restore in submit scripts over hand-loading a long list — it’s reproducible and self-documents the environment your job needs.

Load modules in your submit script, not just interactively

Modules loaded on the login node don’t carry into a batch job. Put the module load / module restore lines inside your sbatch script so the job builds the same environment every time (Section 5.2).

When a package isn’t available as a module — or you need a pip/conda world — see the next section (Section 4.2).

4.2 Conda and containers

4.2.1 Point conda away from home first

The miniconda3 module gives you conda:

module load miniconda3/24.3.0

By default, conda writes environments and a package cache into ~/.conda and ~/miniconda3, inside your 50 GB home quota. A couple of scientific environments will fill it. Before creating any environments, point conda at your project’s 250 GB software space.

Create ~/.condarc:

pkgs_dirs:
  - /project/<project>/software/<you>/conda/pkgs
envs_dirs:
  - /project/<project>/software/<you>/conda/envs

Now new environments and the download cache land in project space:

conda create -n geo python=3.12 numpy xarray netcdf4
conda activate geo
Set .condarc before your first conda create

Environments created before you set .condarc are stranded in home. If you’ve already made the mess: conda env list, remove the home-based envs, and recreate them under project space.

Other conda notes:

  • No mamba is installed on the cluster; use conda directly.
  • Activate conda inside your submit script, not just interactively, so batch jobs get the same environment (Section 5.2).
  • conda clean -a reclaims space from the package cache.

4.2.2 Apptainer: containers for reproducibility and portability

A container bundles an application with its entire operating-system environment into a single image file. On HPC the runtime is Apptainer (formerly Singularity). It runs containers as you, with no root, which makes it the HPC-safe alternative to Docker.

module load apptainer/1.4.1

# Pull a Docker image and convert it to an Apptainer image (.sif):
apptainer pull mytool.sif docker://ghcr.io/someorg/mytool:latest

# Run something inside it:
apptainer exec mytool.sif mytool --help

# Bind your data into the container so it can see it:
apptainer exec --bind /project/<project>:/data mytool.sif mytool /data/input.nc

Store .sif images in /project/<project>/software — they can be large.

4.2.2.1 Container vs. module vs. conda

Situation Best choice
The software is already an Lmod module module (Section 4.1) — least effort
Pure Python/R stack, you want to iterate conda
Complex native dependencies that fight the system Apptainer
You need bit-for-bit reproducibility across machines Apptainer
A published tool ships a Docker image Apptainer (apptainer pull docker://…)
GPUs inside containers

To use the node’s GPUs from inside a container, add the --nv flag so Apptainer exposes the NVIDIA driver: apptainer exec --nv mytool.sif .... Pair this with a GPU job (Section 7.3.2).

4.3 Keep package installs out of home

Home is only 50 GB (Section 3.1), and every language’s package manager dumps its downloads, compiled caches, and installed libraries into a hidden directory in home by default — a few environments and it’s full, at which point jobs and logins start failing in confusing ways. The fix is the same for all three languages: point the package directory at your project’s 250 GB software space, set it once in ~/.bashrc, and create it before you install anything. (Large regenerable caches could live on /gscratch instead, but it’s purged after 90 days — project space is the safer home for software.)

For conda, redirect environments and the package cache via ~/.condarc — see the conda section above (Section 4.2). For plain pip, redirect the cache and build virtual environments in project space rather than ~/.local:

# in ~/.bashrc
export PIP_CACHE_DIR=/project/<project>/software/$USER/pip-cache

# make a venv in project space (load a python module first)
module load gcc/14.2.0 python/3.12.0
python3 -m venv /project/<project>/software/$USER/venvs/myenv
source /project/<project>/software/$USER/venvs/myenv/bin/activate
pip install numpy xarray        # installs into the venv, not home

Avoid pip install --user — it writes to ~/.local and fills home.

R installs user packages into ~/R/... by default (confirmed on the machine). Point R_LIBS_USER at project space and create the directory:

# in ~/.bashrc
export R_LIBS_USER=/project/<project>/software/$USER/Rlibs
mkdir -p "$R_LIBS_USER"

Now install.packages("data.table") — after module load gcc/15.2.0 r/4.5.3 — installs there. For per-project, reproducible libraries, the renv package works well and keeps each project’s packages inside the project directory.

Julia’s depot (~/.julia — downloaded packages, registries, artifacts, and the precompiled cache) is the biggest offender; it routinely grows to many gigabytes. Move it with JULIA_DEPOT_PATH:

# in ~/.bashrc
export JULIA_DEPOT_PATH=/project/<project>/software/$USER/julia_depot

Then module load julia/1.11.6 and ] add DataFrames populates the project-space depot instead of home. Set this before your first julia run; if you already have a large ~/.julia, move it (mv ~/.julia "$JULIA_DEPOT_PATH") and set the variable.

After editing ~/.bashrc, run source ~/.bashrc (or reconnect) to apply it, and put the same export and module load lines in your submit scripts so batch jobs use the same locations (Section 5.2).

Next: running your work through the scheduler (Section 5.1).