nb2slurm on Windows: installing WSL¶
nb2slurm talks to your HPC cluster in two ways, and only one of them works natively on Windows.
Submitting jobs, checking status, cancelling jobs (wf.submit(), wf.status(), wf.cancel(), building conda environments) all go over SSH via paramiko, a pure-Python SSH client. This works fine on plain Windows.
Copying your notebooks up and results back down (wf.push(), wf.pull()) shells out to the real rsync command-line tool, which Windows does not ship. This notebook installs WSL (Windows Subsystem for Linux) and rsync inside it, so push/pull work.
Why WSL specifically? Almost every HPC cluster runs Linux end-to-end - SLURM, conda, the batch scripts nb2slurm generates. paramiko hides that gap for SSH commands, but rsync is a local dependency: a real program that has to run on your machine. WSL is the officially supported, least fragile way to get one on Windows. See hpc-for-beginners.md for what happens on the cluster side.
Important limitations of this notebook:
Installing WSL requires Administrator privileges. If the check below says you’re not elevated, close this and reopen Jupyter/your terminal as Administrator, then re-run.
Installing WSL requires a restart. After the reboot, come back to this notebook and re-run it from the top - the earlier steps will detect WSL is already installed and skip ahead automatically.
The first time a fresh WSL distro boots, Windows opens a console window asking you to pick a Linux username/password. That step is interactive by design (it’s creating your Linux account) and can’t be scripted from here - just fill it in once when it appears.
import ctypes
import platform
import subprocess
IS_WINDOWS = platform.system() == "Windows"
if not IS_WINDOWS:
print(
f"This notebook is for Windows; detected {platform.system()}. Nothing to do here - "
"rsync is already available on macOS/Linux, install it with your usual package manager "
"if it's missing (e.g. `apt install rsync` / `brew install rsync`)."
)
Step 1 - check administrator privileges¶
Installing WSL requires an elevated (Administrator) process. This only checks; it can’t elevate itself for you.
def is_admin() -> bool:
if not IS_WINDOWS:
return False
try:
return bool(ctypes.windll.shell32.IsUserAnAdmin())
except Exception:
return False
if IS_WINDOWS:
if is_admin():
print("Running as Administrator - good, you can proceed.")
else:
print(
"NOT running as Administrator.\n"
"Close this Jupyter server, reopen your terminal (PowerShell / Anaconda Prompt / etc.) "
"via 'Run as administrator', restart Jupyter from there, and re-run this notebook."
)
Step 2 - check whether WSL is already installed¶
wsl_already_installed = False
if IS_WINDOWS:
try:
result = subprocess.run(["wsl", "--status"], capture_output=True, text=True)
print(result.stdout or result.stderr)
wsl_already_installed = result.returncode == 0
except FileNotFoundError:
print(
"The 'wsl' command was not found. This Windows build may be too old for "
"'wsl --install'; see https://learn.microsoft.com/windows/wsl/install-manual"
)
Step 3 - install WSL¶
Skips automatically if Step 2 already found an installation.
if IS_WINDOWS and not wsl_already_installed:
if not is_admin():
print(
"Skipping install: re-run this notebook as Administrator first (see Step 1)."
)
else:
print("Installing WSL - this can take a few minutes...")
result = subprocess.run(["wsl", "--install"], capture_output=True, text=True)
print(result.stdout)
print(result.stderr)
print(
"\nWSL install triggered. RESTART YOUR COMPUTER now, then come back to this "
"notebook and re-run all cells from the top - Step 2 will detect WSL is "
"installed and the rest of the notebook will continue."
)
elif IS_WINDOWS:
print("WSL is already installed - continuing to the next step.")
Step 4 - first-time Ubuntu setup (interactive, one-time)¶
If Step 3 just installed WSL for the first time, a console window pops up automatically the first time WSL boots after your restart, asking you to create a Linux username and password. This is separate from your Windows login and can’t be scripted from a notebook cell - fill it in once in that window, then continue below.
Step 5 - install rsync and openssh-client inside WSL¶
Runs as root inside WSL so there’s no interactive sudo password prompt to get stuck on.
rsync_ready = False
if IS_WINDOWS:
print("Installing rsync + openssh-client inside WSL...")
install_cmd = [
"wsl",
"-u",
"root",
"bash",
"-lc",
"apt-get update && apt-get install -y rsync openssh-client",
]
result = subprocess.run(install_cmd, capture_output=True, text=True)
print(result.stdout[-2000:])
print(result.stderr[-2000:])
if result.returncode != 0:
print(
"\nThis usually means WSL has no Linux distro registered yet - finish Step 4 "
"(the one-time Ubuntu username/password setup) and re-run this cell."
)
else:
rsync_ready = True
Step 6 - verify¶
if IS_WINDOWS:
result = subprocess.run(["wsl", "which", "rsync"], capture_output=True, text=True)
path = result.stdout.strip()
if result.returncode == 0 and path:
print(
f"rsync is installed at {path} inside WSL. push()/pull() should work now."
)
else:
print(
"rsync was not found inside WSL - re-run Step 5, or see the troubleshooting note above."
)
Using rsync from your Windows Python environment¶
nb2slurm’s push/pull call the rsync command directly as a subprocess - not through wsl rsync .... The most reliable setup is to run your notebooks/Python from inside WSL (install conda + nb2slurm there), so rsync is just on the PATH. Calling the WSL-installed rsync from a Windows-native Python process needs extra PATH/shim wiring and is more fragile.
If you’d rather avoid WSL entirely, any Windows build of rsync on your PATH works (Git for Windows’ optional Unix tools, MSYS2, Cygwin) - WSL is simply the option Microsoft supports directly and least likely to give you path-translation surprises talking to a real Linux ssh.