JASMIN Help Site logo JASMIN Help Site logo
  • Docs 
  • Guides 
  • Training 
  • Discussions   

  •   Search this site  

Can't find what you're looking for?

Try our Google custom search, across all JASMIN sites

Docs
  • getting started
    • get started with jasmin
    • generate ssh key pair
    • get jasmin portal account
    • get login account
    • beginners training workshop
    • how to contact us about jasmin issues
    • jasmin status
    • jasmin training accounts
    • tips for new users
    • how to login
    • multiple account types
    • present ssh key
    • reconfirm email address
    • reset jasmin account password
    • ssh auth
    • storage
    • understanding new jasmin storage
    • update a jasmin account
  • interactive computing
    • interactive computing overview
    • check network details
    • login servers
    • login problems
    • graphical linux desktop access using nx
    • sci servers
    • tenancy sci analysis vms
    • transfer servers
    • jasmin notebooks service
    • jasmin notebooks service with gpus
    • creating a virtual environment in the notebooks service
    • project specific servers
    • dask gateway
    • access from vscode
  • batch computing
    • lotus overview
    • slurm scheduler overview
    • slurm queues
    • lotus cluster specification
    • how to monitor slurm jobs
    • how to submit a job
    • how to submit an mpi parallel job
    • example job 2 calc md5s
    • orchid gpu cluster
    • slurm status
    • slurm quick reference
  • software on jasmin
    • software overview
    • quickstart software envs
    • python virtual environments
    • additional software
    • community software esmvaltool
    • community software checksit
    • compiling and linking
    • conda environments and python virtual environments
    • conda removal
    • creating and using miniforge environments
    • idl
    • jasmin sci software environment
    • jasmin software faqs
    • jaspy envs
    • matplotlib
    • nag library
    • name dispersion model
    • geocat replaces ncl
    • postgres databases on request
    • running python on jasmin
    • running r on jasmin
    • rocky9 migration 2024
    • share software envs
  • data transfer
    • data transfer overview
    • data transfer tools
    • globus transfers with jasmin
    • bbcp
    • ftp and lftp
    • globus command line interface
    • globus connect personal
    • gridftp ssh auth
    • rclone
    • rsync scp sftp
    • scheduling automating transfers
    • transfers from archer2
  • short term project storage
    • apply for access to a gws
    • elastic tape command line interface hints
    • faqs storage
    • gws etiquette
    • gws scanner ui
    • gws scanner
    • gws alert system
    • install xfc client
    • xfc
    • introduction to group workspaces
    • jdma
    • managing a gws
    • secondary copy using elastic tape
    • share gws data on jasmin
    • share gws data via http
    • using the jasmin object store
    • configuring cors for object storage
  • long term archive storage
    • ceda archive
  • mass
    • external access to mass faq
    • how to apply for mass access
    • moose the mass client user guide
    • setting up your jasmin account for access to mass
  • for cloud tenants
    • introduction to the jasmin cloud
    • jasmin cloud portal
    • cluster as a service
    • cluster as a service kubernetes
    • cluster as a service identity manager
    • cluster as a service slurm
    • cluster as a service pangeo
    • cluster as a service shared storage
    • adding and removing ssh keys from an external cloud vm
    • provisioning tenancy sci vm managed cloud
    • sysadmin guidance external cloud
    • best practice
  • workflow management
    • rose cylc on jasmin
    • using cron
  • uncategorized
    • mobaxterm
    • requesting resources
    • processing requests for resources
    • acknowledging jasmin
    • approving requests for access
    • working with many linux groups
    • jasmin conditions of use
  • getting started
    • get started with jasmin
    • generate ssh key pair
    • get jasmin portal account
    • get login account
    • beginners training workshop
    • how to contact us about jasmin issues
    • jasmin status
    • jasmin training accounts
    • tips for new users
    • how to login
    • multiple account types
    • present ssh key
    • reconfirm email address
    • reset jasmin account password
    • ssh auth
    • storage
    • understanding new jasmin storage
    • update a jasmin account
  • interactive computing
    • interactive computing overview
    • check network details
    • login servers
    • login problems
    • graphical linux desktop access using nx
    • sci servers
    • tenancy sci analysis vms
    • transfer servers
    • jasmin notebooks service
    • jasmin notebooks service with gpus
    • creating a virtual environment in the notebooks service
    • project specific servers
    • dask gateway
    • access from vscode
  • batch computing
    • lotus overview
    • slurm scheduler overview
    • slurm queues
    • lotus cluster specification
    • how to monitor slurm jobs
    • how to submit a job
    • how to submit an mpi parallel job
    • example job 2 calc md5s
    • orchid gpu cluster
    • slurm status
    • slurm quick reference
  • software on jasmin
    • software overview
    • quickstart software envs
    • python virtual environments
    • additional software
    • community software esmvaltool
    • community software checksit
    • compiling and linking
    • conda environments and python virtual environments
    • conda removal
    • creating and using miniforge environments
    • idl
    • jasmin sci software environment
    • jasmin software faqs
    • jaspy envs
    • matplotlib
    • nag library
    • name dispersion model
    • geocat replaces ncl
    • postgres databases on request
    • running python on jasmin
    • running r on jasmin
    • rocky9 migration 2024
    • share software envs
  • data transfer
    • data transfer overview
    • data transfer tools
    • globus transfers with jasmin
    • bbcp
    • ftp and lftp
    • globus command line interface
    • globus connect personal
    • gridftp ssh auth
    • rclone
    • rsync scp sftp
    • scheduling automating transfers
    • transfers from archer2
  • short term project storage
    • apply for access to a gws
    • elastic tape command line interface hints
    • faqs storage
    • gws etiquette
    • gws scanner ui
    • gws scanner
    • gws alert system
    • install xfc client
    • xfc
    • introduction to group workspaces
    • jdma
    • managing a gws
    • secondary copy using elastic tape
    • share gws data on jasmin
    • share gws data via http
    • using the jasmin object store
    • configuring cors for object storage
  • long term archive storage
    • ceda archive
  • mass
    • external access to mass faq
    • how to apply for mass access
    • moose the mass client user guide
    • setting up your jasmin account for access to mass
  • for cloud tenants
    • introduction to the jasmin cloud
    • jasmin cloud portal
    • cluster as a service
    • cluster as a service kubernetes
    • cluster as a service identity manager
    • cluster as a service slurm
    • cluster as a service pangeo
    • cluster as a service shared storage
    • adding and removing ssh keys from an external cloud vm
    • provisioning tenancy sci vm managed cloud
    • sysadmin guidance external cloud
    • best practice
  • workflow management
    • rose cylc on jasmin
    • using cron
  • uncategorized
    • mobaxterm
    • requesting resources
    • processing requests for resources
    • acknowledging jasmin
    • approving requests for access
    • working with many linux groups
    • jasmin conditions of use
  1.   Batch Computing
  1. Home
  2. Docs
  3. Batch Computing
  4. How to submit a job

How to submit a job

 

Share via
JASMIN Help Site
Link copied to clipboard

How to submit a batch job to Slurm

On this page
What is a batch job?   Which servers can you submit jobs from?   Method 1: Submit via a Slurm job script   New Slurm job accounting hierarchy   Partitions and QoS   Method 2: Submit via command-line options   Method 3: Submit an interactive session via salloc   Interactive execution with pseudo-shell terminal on the compute LOTUS node   Job array submission  

What is a batch job?  

A batch job is a task that, once submitted to the scheduler, can run without further interaction from the user. A user writes a script containing both the command(s) to be run and directives for the scheduler as to how the job should be run. The batch system then selects the resources required by the job and decides when and where to run the job. Note: the term “job” is used throughout this documentation to mean a “batch job”.

There are two ways of submitting a job to Slurm:

  1. Submit via a Slurm job script - create a bash script that includes directives to the Slurm scheduler
  2. Submit via command-line options - provide directives to Slurm via command-line arguments

Both options are described below.

Which servers can you submit jobs from?  

Jobs can be submitted to Slurm from any of the sci servers. Check the current list of servers on that page.

Method 1: Submit via a Slurm job script  

The Slurm job submission command is:

sbatch myjobscript

The job script is a Bash script of user’s application and includes a list of Slurm directives, prefixed with #SBATCH as shown in this example:

 
Remove any trailing whitespace
#!/bin/bash
#SBATCH --job-name="My test job"
#SBATCH --time=00:01:00
#SBATCH --mem=1M
#SBATCH --account=mygws
#SBATCH --partition=debug
#SBATCH --qos=debug
#SBATCH -o %j.out
#SBATCH -e %j.err

# executable
sleep 5s

Explanation:

Submitting the above script (if you had access to the mygws account) creates a job named My test job with an estimated run time of 00:01:00 (1 minute), memory requirement of 1M (1 Megabyte), run against the account mygws on the partition debug using Qos debug, and writing its STDOUT (standard output) to file %j.out and its STDERR to file %j.err (where %j represents the job ID that the scheduler assigns to the job).

The task itself is the command sleep 5s which just pauses for 5 seconds before exiting. This is what you would replace with your actual processing command(s), so you need to have an idea of how long it will take to run (TIP: run it manually first with time <cmd> to find out!)

For details of other submission parameters, see job specification.

New Slurm job accounting hierarchy  

Slurm accounting by project has been introduced as a means of monitoring compute usage by projects on JASMIN. These projects align with group workspaces (GWSs), and you will automatically be added to Slurm accounts corresponding to any GWS projects that you belong to.

To find what Slurm accounts and quality of services (QoS) that you have access to, use the useraccounts command on any sci machine. Output should be similar to one or more of the lines below.

useraccounts
# sacctmgr show user fred withassoc format=user,account,qos%-50
User       Account        QOS
---------- -------------- -------------------------------------
      fred  mygws         debug,high,long,short,standard
      fred  orchid        debug,high,long,short,standard

You should use the relevant account for your project’s task with the --account directive in your job script.

Users who do not belong to any group workspaces will be assigned the no-project account and should use that in their job submissions. Please ignore and do not use the group shobu.

Partitions and QoS  

There are two partitions currently available on LOTUS, with associated allowed quality of service (QoS) as shown below:

Partition Allowed QoS
standard standard, short, long, high
debug debug, reservation
QoS Priority Max CPUs per job Max wall time
standard 500 1 24 hours
short 550 1 4 hours
long 350 1 5 days
high 450 96 2 days
debug 500 8 1 hour

Once you’ve chosen the partition and QoS you need, provide the partition in the --partition directive and the QoS in the --qos directive.

Method 2: Submit via command-line options  

If you have an existing script, written in any language, that you wish to submit to LOTUS then you can do so by providing Slurm directives as command-line arguments. For example, if you have a script my-script.py that takes a single argument -f <filepath>, you can submit it using sbatch as follows:

sbatch -A mygws -p debug -q debug -t 03:00 -o job01.out -e job01.err my-script.py -f myfile.txt

This approach allows you to submit jobs without writing additional job scripts to wrap your existing code.

Check the official documentation for sbatch, its arguments and their syntax here  .

Method 3: Submit an interactive session via salloc  

Testing a job on LOTUS can be carried out in an interactive manner by obtaining a Slurm job allocation or resources (a set of nodes) via the Slurm command salloc. The code/application is executed and the allocation is released after a specific time - default 30 mins - when the testing is finished.

Interactive execution with pseudo-shell terminal on the compute LOTUS node  

The job is executed on the LOTUS compute node by allocating resources with salloc. See the example below:

salloc -p standard -q high -A mygws --ntasks-per-node=2
salloc: Pending job allocation 23506
salloc: job 23506 queued and waiting for resources
salloc: job 23506 has been allocated resources
salloc: Granted job allocation 23506
salloc: Nodes host580 are ready for job

Official documentation for the salloc command is available here  .

At this point, your shell prompt will change to the LOTUS compute node. You will have the allocated compute that you requested at this shell. For example the command hostname is executed twice as there are 2 CPUs and each outputs the name of the node:

srun hostname
host580.jc.rl.ac.uk
host580.jc.rl.ac.uk

Official documentation for the srun command is available here  .

The job allocation ID 23506 has 2 CPUs on the compute node host580 and can be checked from another terminal as shown below:

squeue -u fred -o"%.18i %.9P %.11j %.8u %.2t %.10M %.6D %.6C %R"
JOBID PARTITION           NAME       USER  ST       TIME  NODES   CPUS NODELIST(REASON)
23506 standard      interactive   fred   R       1:32      1      2 host580

 
squeue --me is equivalent to squeue -u fred
 

Please DO NOT use watch or equivalent polling utilities with Slurm as they are wasteful of resources and cause communication issues for the scheduler.

Your process may be killed if this is detected.

Official documentation for the squeue command is available here  .

Once you’re finished, type exit to relinquish the allocation. This will happen automatically once the time limit on the job runs out.

Job array submission  

Job arrays are groups of jobs with the same executable and resource requirements, but different input files. Job arrays can be submitted, controlled, and monitored as a single unit or as individual jobs or groups of jobs. Each job submitted from a job array shares the same job ID as the job array and is uniquely referenced using an array index. This approach is useful for ‘high throughput’ tasks, for example where you want to run your simulation with different driving data or run the same processing task on multiple data files.

Important note: The maximum job array size that Slurm is configured for is MaxArraySize = 10000. If a Job array of size is greater than 10000 is submitted, Slurm will reject the job submission with the following error message: “Job array index too large. Job not submitted.”

Taking a simple R submission script as an example:

#!/bin/bash 
#SBATCH --job-name=myRtest
#SBATCH --time=30:00
#SBATCH --account=mygws  
#SBATCH --partition=debug
#SBATCH --qos=debug      
#SBATCH -o %j.out 
#SBATCH -e %j.err 

module add jasr
Rscript TestRFile.R dataset1.csv

If you want to run the same script TestRFile.R with input file dataset2.csv through to dataset10.csv, you could create and submit a job script for each dataset. However, by setting up an array job, you could create and submit a single job script.

The corresponding job array script to process 10 input files in a single job submission would look something like this:

#!/bin/bash 
#SBATCH --job-name=myRarray
#SBATCH --time=30:00
#SBATCH --account=mygws  
#SBATCH --partition=debug
#SBATCH --qos=debug
#SBATCH -o %A_%a.out
#SBATCH -e %A_%a.err
#SBATCH --array=1-10

module add jasr
Rscript TestRFile.R datset${SLURM_ARRAY_TASK_ID}.csv

Here the important differences are :

  • The array is created by Slurm directive --array=1-10 by including elements numbered [1-10] to represent our 10 variations
  • The error and output file have the array index %a included in the name and %A is the job ID.
  • The environment variable $SLURM_ARRAY_TASK_ID in the Rscript command is expanded to give the job index

When the job is submitted, Slurm will create 10 tasks under the single job ID. The job array script is submitted in the usual way:

sbatch myRarray.sbatch

If you use the squeue -u <username> command to list your active jobs, you will see 10 tasks with the same Job ID. The tasks can be distinguished by the [index] e.g. jobID_index. Note that individual tasks may be allocated to a range of different hosts on LOTUS.

Last updated on 2025-05-09 as part of:  Update to How to submit a job page (73e77acf9)
On this page:
What is a batch job?   Which servers can you submit jobs from?   Method 1: Submit via a Slurm job script   New Slurm job accounting hierarchy   Partitions and QoS   Method 2: Submit via command-line options   Method 3: Submit an interactive session via salloc   Interactive execution with pseudo-shell terminal on the compute LOTUS node   Job array submission  
Follow us

Social media & development

   

Useful links

  • CEDA Archive 
  • CEDA Catalogue 
  • JASMIN 
  • JASMIN Accounts Portal 
  • JASMIN Projects Portal 
  • JASMIN Cloud Portal 
  • JASMIN Notebooks Service 
  • JASMIN Community Discussions 

Contact us

  • Helpdesk
UKRI/STFC logo
UKRI/NERC logo
NCAS logo
NCEO logo
Accessibility | Terms and Conditions | Privacy and Cookies
Copyright © 2025 Science and Technology Facilities Council.
Hinode theme for Hugo licensed under Creative Commons (CC BY-NC-SA 4.0).
JASMIN Help Site
Code copied to clipboard