nb2slurm

nb2slurm takes a notebook workflow that runs for one subject (one catchment, one region, one number, …) and generates everything needed to run it for many subjects on a SLURM HPC — driven entirely from a notebook, no command line.

Pre-1.0

nb2slurm works end to end and is released on PyPI, but it is young: the API can still change in a minor release. Pin the version you depend on.

It is both a package and a scaffolder:

• Package (logic): the importable nb2slurm — the Workflow class plus helpers for settings, done-tracking, environments and an SSH transport.

• Scaffolder (templates): bundled Jinja2 templates that Workflow.build() renders into a concrete scripts/ directory for your project.

nb2slurm is the reusable generalisation of the eWaterCycle CCI-analysis-seamless project (its hardcoded cci.py + run_cci.slurm + submit_*.sh).

Where to start

• New to HPC, SLURM or conda? Read HPC for notebook users first.

• Want the short version? Quickstart.

• Converting notebooks you already have? The notebook contract.

• Want to see it end to end? Monte Carlo example.

Getting started

• Installation
  • What you need besides the package
  • Development install
• Quickstart
  • 1. Describe the workflow
  • 2. Build, and prove it on one subject
  • 3. Point at the cluster
  • 4. Submit and watch
  • 5. Get the results back
  • Where the jobs come from
• HPC for notebook users (no Linux required)
  • The one-sentence summary
  • The pieces
  • What you need before you start
  • How nb2slurm hides all of this
  • Mini-glossary
• nb2slurm on Windows: installing WSL
  • Step 1 - check administrator privileges
  • Step 2 - check whether WSL is already installed
  • Step 3 - install WSL
  • Step 4 - first-time Ubuntu setup (interactive, one-time)
  • Step 5 - install rsync and openssh-client inside WSL
  • Step 6 - verify
  • Using rsync from your Windows Python environment

Guide

• The notebook contract
  • 1. The first notebook writes the settings
  • 2. Every later notebook reads it
  • Laptop vs. cluster: on_hpc()
  • Converting a local-only workflow
• jobs.json — jobs and the output tree in one file
  • Format rules
  • Using it
• Environments and kernels
  • 1. Let nb2slurm create the environment
  • 2. Use an environment the cluster already has
  • 3. Different environments for different notebooks
  • Checking before you submit
• Generated files
• Connecting to the cluster
  • Getting a key registered
  • Passphrase-protected keys
• Moving files: push up, pull down
• Control notebooks
  • One source of truth
  • The loop afterwards

Examples

• nb2slurm — how it works (illustrative walkthrough)
  • 0. Install (once)
  • Your settings — edit this one cell
  • 1. The jobs file — one nested JSON drives everything
  • 2. The project layout nb2slurm assumes
  • 3. The notebook contract
  • 4. Describe the environment your code needs (optional)
  • 5. Describe the workflow
  • 6. Generate the SLURM / runner scripts
  • 7. Build the output directory tree (optional)
  • 8. Prove it works on ONE job, locally
  • 9. Connect to the HPC
  • 10. Create the environment on the cluster (one-time, optional)
  • 11. Preflight check
  • 12. Submit the jobs
  • 13. Monitor
  • 14. Cancel (if needed)
  • 15. Re-run safely
• Monte Carlo π example

Reference

• API reference
  • Workflow
  • Environment
  • SSH
  • Inside your notebooks
  • Jobs, outputs and saved config