Making your notebooks nb2slurm-compatible

You already have a notebook workflow that runs for one subject on your laptop, top to bottom. To let nb2slurm run it unattended for many subjects on SLURM — while it still runs locally, unchanged — you adjust the notebooks in a few specific ways.

This guide is the distilled diff between a local-only tutorial workflow (ewatercycle-climatechangeimpact) and its SLURM-ready twin (CCI-analysis-seamless): the changes that make a notebook safely executable by a machine instead of a human clicking through it.

The running example uses a hydrology workflow keyed by country + region_id.

What nb2slurm does for you (so you don’t)

The whole orchestration layer is generated — you never write it:

  • the papermill runner that chains your notebooks,

  • the job.slurm file (resources, conda, mounts),

  • the submit/cancel scripts and the per-job output directories,

  • a done.csv ledger so finished jobs are skipped on re-run.

nb2slurm also injects parameters for you: it passes the varying value(s) plus outdir into your first notebook, and settings_path into every later one, and it runs everything from the project root with a unique outdir per job.

So you do not hand-write a runner, recompute settings_path per machine, or namespace output folders by hand. What’s left is the handful of in-notebook changes below.

Change 1 — add a parameters-tagged cell (the one that’s mandatory)

papermill (which nb2slurm drives) overrides exactly one cell per notebook: the one tagged parameters. Without it there is nothing to inject, so this change must live in the notebook itself.

In Jupyter: select the cell → Property Inspector (gear icon) → Cell tags → add parameters. (A Jupyter feature, not a nb2slurm one.)

First notebook — the varying value(s) + outdir (the values here are just defaults for local runs):

# --- cell tagged "parameters" ---
country   = "australia"
region_id = "camelsaus_102101A"
outdir    = "."               # nb2slurm passes the per-job output dir on the cluster

Every later notebook — just where to find the settings:

# --- cell tagged "parameters" ---
settings_path = "settings.json"   # nb2slurm passes the real path on the cluster

Change 2 — refer to the parameter variable, never re-hardcode it

The default in the parameters cell (e.g. region_id = "camelsaus_102101A") is intentional and good — it’s the value the local run uses, and the exact thing papermill replaces on SLURM. The mistake to avoid is writing that identifier again as a separate literal somewhere downstream: then the injected value and the hardcoded copy disagree, and your batch jobs all silently process the same region.

So anything that varies per subject must trace back to the parameters-cell variable, never a fresh literal.

DO NOT DO — a second, hardcoded copy papermill can’t reach:

settings["caravan_id"] = "camelsaus_102101A"   # literal; ignores the injected region_id

DO — derive it from the parameter:

settings["caravan_id"] = region_id              # tracks whatever papermill injects

Then the first notebook hands everything forward in one settings file, and the later notebooks read it back:

# first notebook
import nb2slurm
nb2slurm.Settings.write(outdir, {"country": country, "region_id": region_id,
                                 "caravan_id": region_id, "outdir": outdir})

# later notebooks
settings = nb2slurm.Settings.load(settings_path)

Change 3 — make machine-specific paths environment-aware (the “duality”)

This is the change that lets the same notebook be both a local run and a SLURM run. nb2slurm already handles outdir and settings_path for you, so you only need to branch on paths that genuinely differ between your laptop and the cluster — shared datasets, model install dirs, raster files, etc.

Detect where you’re running with nb2slurm.on_hpc() and pick the path accordingly. It checks the environment variables SLURM sets in every job (plus the NB2SLURM sentinel nb2slurm exports), so it works for any user on any cluster.

import nb2slurm
from pathlib import Path

if nb2slurm.on_hpc():
    data_dir           = Path("/project/ewater/Data")  # project based storage
else:
    data_dir           = Path.home() / "/path/to/data/locally"  # Linux

Change 4 — write all outputs under outdir

nb2slurm gives every job its own outdir (e.g. output/australia/camelsaus_102101A/), so parallel jobs never collide — as long as you write everything under it.

Change 5 — make each step idempotent (load-or-generate)

A batch job that re-runs over existing data must not crash or recompute for hours. Tutorials often have a generate(...) cell and a commented-out load(...) you swap by hand; instead, try to load and fall back to generating.

Before

params = run_calibration(...)        # always recomputes; manual load() is commented out

After

params_file = Path(settings["outdir"]) / "params.csv"
try:
    params = load_params(params_file)         # reuse if it's already there
except FileNotFoundError:
    params = run_calibration(...)             # ... otherwise compute and save
    save_params(params, params_file)

(This is per-step resume inside a job; nb2slurm’s done.csv separately skips whole subjects that already finished.)

Change 6 — make execution headless-safe

No human is watching, and the HPC environment is prebuilt:

  • Use display(obj) instead of print(obj) for rich objects (DataFrames, maps), so they render correctly when papermill executes the notebook.

  • Guard interactive !pip install so it’s skipped on the cluster’s ready-made env — again using nb2slurm.on_hpc() rather than a home-path check:

import nb2slurm
run_pips = not nb2slurm.on_hpc()   # only install locally; the HPC env is prebuilt

if run_pips:
    !pip install sceua

Inline installs are fine when you’re clicking through interactively, but wasteful or breaking under batch. If these installs are needed locally, I suggest you make a separate environment on the HPC.

Change 7 — import helpers from scripts/ robustly

Keep notebooks/ clean: put reusable functions in scripts/ and import them. Make the import work whether the working directory is notebooks/ (local) or the project root (on the HPC). On the cluster the job runs from the project root, so the import just works; run locally from notebooks/, add the project root to the path. nb2slurm.on_hpc() tells the two apart:

import sys
from pathlib import Path
import nb2slurm

if not nb2slurm.on_hpc():
    sys.path.append(str(Path().resolve().parent))   # project root -> scripts/ imports
from scripts.helpers import load_region

A try/except ImportError that appends the project root works too — the key point is to add the parent (project root), never sys.path.append("."), which only adds notebooks/ and won’t find scripts/.

Checklist

For each notebook in your workflow:

  • [ ] 1. Add a parameters-tagged cell — first notebook: varying value(s) + outdir; later notebooks: settings_path.

  • [ ] 2. Replace hardcoded per-subject constants with the injected parameters; first notebook Settings.write(...), later notebooks Settings.load(...).

  • [ ] 3. Branch machine-specific paths on nb2slurm.on_hpc(), not a hardcoded location or a Path.home() string match.

  • [ ] 4. Write every output under settings["outdir"].

  • [ ] 5. Make expensive steps load-or-generate so re-runs are cheap and safe.

  • [ ] 6. display() not print() for rich objects; guard !pip install with nb2slurm.on_hpc().

  • [ ] 7. Import helpers from scripts/ by adding the project root to the path when running locally (if not nb2slurm.on_hpc(): sys.path.append(str(Path().resolve().parent))), never sys.path.append(".").

Once your notebooks satisfy this, define your subjects in jobs.json and drive the run from the control notebooks in docs/control/.