A crash-course in using SLURM

Slurm is a workload manager for distributed systems.

The most important commands

squeue -u USERNAME
squeue -p PARTITION
sbatch SCRIPT
scancel SLURM_ID

squeue -u USERNAME will tell you what jobs are running for the user. You can keep track of your jobs this way, although it only tells you which ones are running. You can look at what is happening (to a degree) by checking the log files which appear in your home directory (or somewhere else if you specify it). If you’re running a python script and want to monitor what’s happening, it’s better to use logging rather than print statements, as the print statements only appear in the log file after the whole script is finished running.

squeue -p PARTITION will show you all the jobs being run on a specific partition. In the case of Saga, we usually stick to normal and accel. Accel has GPUs, so if you’re running a compute heavy job, it’s probably better.

sbatch SCRIPT will run a slurm job that you have set up. This also give you the SLURM_ID for the job. If you need to cancel the job for some reason, run scancel SLURM_ID

Importing / Setting up necessary libraries and dependencies

Saga already has most of the dependencies you need installed (Pytorch, Sklearn, etc). You can check available modules here.

To upload a module, either in an interactive session (when you first log on), do:

module load MODULENAME

Example slurm job

#!/bin/bash
#
#SBATCH --job-name=bert_finetune --account=nn9447k  #change the job-name
#SBATCH --output=bert_finetune-%j.out               #name of log file
#
#SBATCH --partition=normal                          #normal or accel
#SBATCH --time=48:00:00                             #how long to run
#
#
# Max memory usage:
#SBATCH --mem-per-cpu=100G
#
# Number of threads per task:
#SBATCH --cpus-per-task=1

## Set up job environtment:
# source /cluster/bin/jobsetup
module restore system                              #clear any inherited modules
set -o errexit                                     #exit on errors

# List imported modules
module list

# Load models
module load Python/3.6.6-fosscuda-2018b

## In addition to the models already available, sometimes I need extra stuff.
## I set up virtual environments and use pip --user install X to get what I
## need.

# load virtual environment
source /cluster/home/jeremycb/doc_level_transfer_hierarchical/my_virtualenv/bin/activate


# Run the python script

python task_tuning.py --data_dir="./data" --bert_model="bert-base-cased" --output_dir="task_tuned/ontonotes-reuters-50000" --trained_model_dir="lm_output/ontonotes-reuters" --do_train --do_eval --save_all_epochs --num_training_examples=50000