How to submit a job
How to submit a batch job to Slurm
A batch job is a task that, once submitted to the scheduler, can run without further interaction from the user. A user writes a script containing both the command(s) to be run and directives for the scheduler as to how the job should be run. The batch system then selects the resources for the job and decides when and where to run the job. Note: the term “job” is used throughout this documentation to mean a “batch job”.
There are two ways of submitting a job to Slurm:
Both options are described below.
Jobs can be submitted to Slurm from any of the sci servers. Check the current list of servers on that page.
The Slurm job submission command is:
sbatch myjobscript
The job script is a Bash script of user’s application and includes a list of
Slurm directives, prefixed with #SBATCH
as shown in this example:
#!/bin/bash
#SBATCH --job-name="My test job"
#SBATCH --time=00:01:00
#SBATCH --mem=1M
#SBATCH --account=mygws
#SBATCH --partition=debug
#SBATCH --qos=debug
#SBATCH -o %j.out
#SBATCH -e %j.err
# executable
sleep 5s
Explanation:
Submitting the above script (if you had access to the mygws
account) creates a job named My test job
with an estimated run time of
00:01:00
(1 minute), memory requirement of 1M
(1 Megabyte), run against the account
mygws
on the partition debug
using Qos debug
, and writing its STDOUT (standard output) to file
%j.out
and its STDERR to file %j.err
(where %j
represents the job ID that the scheduler assigns to the job).
The task itself is the command sleep 5s
which just pauses for 5 seconds before exiting. This is what you would replace
with your actual processing command(s), so you need to have an idea of how long it will take to run (TIP: run it manually first with time <cmd>
to find out!)
For details of other submission parameters, see job specification.
Slurm accounting by project has been introduced as a means of monitoring compute usage by projects on JASMIN. These projects align with group workspaces (GWSs), and you will automatically be added to Slurm accounts corresponding to any GWS projects that you belong to.
To find what Slurm accounts and quality of services (QoS) that you have access to, use the useraccounts
command on any sci
machine.
Output should be similar to one or more of the lines below.
useraccounts
# sacctmgr show user fred withassoc format=user,account,qos%-50
User Account QOS
---------- -------------- -------------------------------------
fred mygws debug,highres,long,short,standard
fred orchid debug,highres,long,short,standard
You should use the relevant account for your project’s task with the --account
directive in your job script.
Users who do not belong to any group workspaces will be assigned the no-project
account and should use that in their job submissions.
Please ignore and do not use the group shobu
.
There are 3 partitions currently available on LOTUS, with associated allowed quality of service (QoS) as shown below:
Partition | Allowed QoS |
---|---|
standard |
standard , short , long |
highres |
highres , reservation |
debug |
debug , reservation |
QoS | Priority | Max CPUs per job | Max wall time |
---|---|---|---|
standard |
500 | 1 | 24 hours |
short |
550 | 1 | 4 hours |
long |
350 | 1 | 5 days |
highres |
450 | 2 days | |
debug |
500 | 1 hour |
Once you’ve chosen the partition and QoS you need, provide the partition in the --partition
directive and the QoS in the --qos
directive.
If you have an existing script, written in any language, that you wish to
submit to LOTUS then you can do so by providing Slurm directives as command-line
arguments. For example, if you have a script my-script.py
that takes a
single argument -f <filepath>
, you can submit it using sbatch
as follows:
sbatch -A mygws -p debug -q debug -t 03:00 -o job01.out -e job01.err my-script.py -f myfile.txt
This approach allows you to submit jobs without writing additional job scripts to wrap your existing code.
Check the official documentation for sbatch
, its arguments and their syntax
here
.
Testing a job on LOTUS can be carried out in an interactive manner by
obtaining a Slurm job allocation or resources (a set of nodes) via the Slurm
command salloc
. The code/application is executed and the allocation is
released after a specific time - default 30 mins - when the testing is finished.
The job is executed on the LOTUS compute node by allocating resources with salloc
.
See the example below:
salloc -p highres -q highres -A mygws --ntasks-per-node=2
salloc: Pending job allocation 23506
salloc: job 23506 queued and waiting for resources
salloc: job 23506 has been allocated resources
salloc: Granted job allocation 23506
salloc: Nodes host580 are ready for job
Official documentation for the salloc
command is available
here
.
At this point, your shell prompt will change to the LOTUS compute node.
You will have the allocated compute that you requested at this shell.
For example the command hostname
is executed twice as there are 2 CPUs
and each outputs the name of the node:
srun hostname
host580.jc.rl.ac.uk
host580.jc.rl.ac.uk
Official documentation for the srun
command is available
here
.
The job allocation ID 23506
has 2 CPUs on the compute node host580
and can be
checked from another terminal as shown below:
squeue -u fred -o"%.18i %.9P %.11j %.8u %.2t %.10M %.6D %.6C %R"
JOBID PARTITION NAME USER ST TIME NODES CPUS NODELIST(REASON)
23506 highres interactive fred R 1:32 1 2 host580
squeue --me
is equivalent to squeue -u fred
Please DO NOT use watch
or equivalent polling utilities with Slurm
as they are wasteful of resources and cause communication issues for the scheduler.
Your process may be killed if this is detected.
Official documentation for the squeue
command is available
here
.
Once you’re finished, type exit
to relinquish the allocation. This will happen
automatically once the time limit on the job runs out.
Job arrays are groups of jobs with the same executable and resource requirements, but different input files. Job arrays can be submitted, controlled, and monitored as a single unit or as individual jobs or groups of jobs. Each job submitted from a job array shares the same job ID as the job array and is uniquely referenced using an array index. This approach is useful for ‘high throughput’ tasks, for example where you want to run your simulation with different driving data or run the same processing task on multiple data files.
Important note: The maximum job array size that Slurm is configured for is
MaxArraySize = 10000
. If a Job array of size is greater than 10000 is
submitted, Slurm will reject the job submission with the following error
message: “Job array index too large. Job not submitted.”
Taking a simple R submission script as an example:
#!/bin/bash
#SBATCH --job-name=myRtest
#SBATCH --time=30:00
#SBATCH --account=mygws
#SBATCH --partition=debug
#SBATCH --qos=debug
#SBATCH -o %j.out
#SBATCH -e %j.err
module add jasr
Rscript TestRFile.R dataset1.csv
If you want to run the same script TestRFile.R
with input file dataset2.csv
through to dataset10.csv
, you could create and submit a job script for each dataset. However, by setting up an array job, you could create and submit a single job script.
The corresponding job array script to process 10 input files in a single job submission would look something like this:
#!/bin/bash
#SBATCH --job-name=myRarray
#SBATCH --time=30:00
#SBATCH --account=mygws
#SBATCH --partition=debug
#SBATCH --qos=debug
#SBATCH -o %A_%a.out
#SBATCH -e %A_%a.err
#SBATCH --array=1-10
module add jasr
Rscript TestRFile.R datset${SLURM_ARRAY_TASK_ID}.csv
Here the important differences are :
--array=1-10
by including elements numbered [1-10]
to represent our 10 variations%a
included in the name and %A
is the job ID.$SLURM_ARRAY_TASK_ID
in the Rscript
command is expanded to give the job indexWhen the job is submitted, Slurm will create 10 tasks under the single job ID. The job array script is submitted in the usual way:
sbatch myRarray.sbatch
If you use the squeue -u <username>
command to list your active jobs, you
will see 10 tasks with the same Job ID. The tasks can be distinguished by the
[index]
e.g. jobID_index. Note that individual tasks may be allocated to a
range of different hosts on LOTUS.