JASMIN Help Site logo JASMIN Help Site logo
  • Docs 
  • Guides 
  • Training 
  • Discussions   

  •   Search this site  

Can't find what you're looking for?

Try our Google custom search, across all JASMIN sites

Docs
  • getting started
    • get started with jasmin
    • generate ssh key pair
    • get jasmin portal account
    • get login account
    • beginners training workshop
    • how to contact us about jasmin issues
    • jasmin status
    • jasmin training accounts
    • tips for new users
    • how to login
    • multiple account types
    • present ssh key
    • reconfirm email address
    • reset jasmin account password
    • ssh auth
    • storage
    • understanding new jasmin storage
    • update a jasmin account
  • interactive computing
    • interactive computing overview
    • check network details
    • login servers
    • login problems
    • graphical linux desktop access using nx
    • sci servers
    • tenancy sci analysis vms
    • transfer servers
    • jasmin notebooks service
    • jasmin notebooks service with gpus
    • creating a virtual environment in the notebooks service
    • project specific servers
    • dask gateway
    • access from vscode
  • batch computing
    • lotus overview
    • slurm scheduler overview
    • slurm queues
    • lotus cluster specification
    • how to monitor slurm jobs
    • how to submit a job
    • how to submit an mpi parallel job
    • example job 2 calc md5s
    • orchid gpu cluster
    • slurm status
    • slurm quick reference
  • software on jasmin
    • software overview
    • quickstart software envs
    • python virtual environments
    • additional software
    • community software esmvaltool
    • community software checksit
    • compiling and linking
    • conda environments and python virtual environments
    • conda removal
    • creating and using miniforge environments
    • idl
    • jasmin sci software environment
    • jasmin software faqs
    • jaspy envs
    • matplotlib
    • nag library
    • name dispersion model
    • geocat replaces ncl
    • postgres databases on request
    • running python on jasmin
    • running r on jasmin
    • rocky9 migration 2024
    • share software envs
  • data transfer
    • data transfer overview
    • data transfer tools
    • globus transfers with jasmin
    • bbcp
    • ftp and lftp
    • globus command line interface
    • globus connect personal
    • gridftp ssh auth
    • rclone
    • rsync scp sftp
    • scheduling automating transfers
    • transfers from archer2
  • short term project storage
    • apply for access to a gws
    • elastic tape command line interface hints
    • faqs storage
    • gws etiquette
    • gws scanner ui
    • gws scanner
    • gws alert system
    • install xfc client
    • xfc
    • introduction to group workspaces
    • jdma
    • managing a gws
    • secondary copy using elastic tape
    • share gws data on jasmin
    • share gws data via http
    • using the jasmin object store
    • configuring cors for object storage
  • long term archive storage
    • ceda archive
  • mass
    • external access to mass faq
    • how to apply for mass access
    • moose the mass client user guide
    • setting up your jasmin account for access to mass
  • for cloud tenants
    • introduction to the jasmin cloud
    • jasmin cloud portal
    • cluster as a service
    • cluster as a service kubernetes
    • cluster as a service identity manager
    • cluster as a service slurm
    • cluster as a service pangeo
    • cluster as a service shared storage
    • adding and removing ssh keys from an external cloud vm
    • provisioning tenancy sci vm managed cloud
    • sysadmin guidance external cloud
    • best practice
  • workflow management
    • rose cylc on jasmin
    • using cron
  • uncategorized
    • mobaxterm
    • requesting resources
    • processing requests for resources
    • acknowledging jasmin
    • approving requests for access
    • working with many linux groups
    • jasmin conditions of use
  • getting started
    • get started with jasmin
    • generate ssh key pair
    • get jasmin portal account
    • get login account
    • beginners training workshop
    • how to contact us about jasmin issues
    • jasmin status
    • jasmin training accounts
    • tips for new users
    • how to login
    • multiple account types
    • present ssh key
    • reconfirm email address
    • reset jasmin account password
    • ssh auth
    • storage
    • understanding new jasmin storage
    • update a jasmin account
  • interactive computing
    • interactive computing overview
    • check network details
    • login servers
    • login problems
    • graphical linux desktop access using nx
    • sci servers
    • tenancy sci analysis vms
    • transfer servers
    • jasmin notebooks service
    • jasmin notebooks service with gpus
    • creating a virtual environment in the notebooks service
    • project specific servers
    • dask gateway
    • access from vscode
  • batch computing
    • lotus overview
    • slurm scheduler overview
    • slurm queues
    • lotus cluster specification
    • how to monitor slurm jobs
    • how to submit a job
    • how to submit an mpi parallel job
    • example job 2 calc md5s
    • orchid gpu cluster
    • slurm status
    • slurm quick reference
  • software on jasmin
    • software overview
    • quickstart software envs
    • python virtual environments
    • additional software
    • community software esmvaltool
    • community software checksit
    • compiling and linking
    • conda environments and python virtual environments
    • conda removal
    • creating and using miniforge environments
    • idl
    • jasmin sci software environment
    • jasmin software faqs
    • jaspy envs
    • matplotlib
    • nag library
    • name dispersion model
    • geocat replaces ncl
    • postgres databases on request
    • running python on jasmin
    • running r on jasmin
    • rocky9 migration 2024
    • share software envs
  • data transfer
    • data transfer overview
    • data transfer tools
    • globus transfers with jasmin
    • bbcp
    • ftp and lftp
    • globus command line interface
    • globus connect personal
    • gridftp ssh auth
    • rclone
    • rsync scp sftp
    • scheduling automating transfers
    • transfers from archer2
  • short term project storage
    • apply for access to a gws
    • elastic tape command line interface hints
    • faqs storage
    • gws etiquette
    • gws scanner ui
    • gws scanner
    • gws alert system
    • install xfc client
    • xfc
    • introduction to group workspaces
    • jdma
    • managing a gws
    • secondary copy using elastic tape
    • share gws data on jasmin
    • share gws data via http
    • using the jasmin object store
    • configuring cors for object storage
  • long term archive storage
    • ceda archive
  • mass
    • external access to mass faq
    • how to apply for mass access
    • moose the mass client user guide
    • setting up your jasmin account for access to mass
  • for cloud tenants
    • introduction to the jasmin cloud
    • jasmin cloud portal
    • cluster as a service
    • cluster as a service kubernetes
    • cluster as a service identity manager
    • cluster as a service slurm
    • cluster as a service pangeo
    • cluster as a service shared storage
    • adding and removing ssh keys from an external cloud vm
    • provisioning tenancy sci vm managed cloud
    • sysadmin guidance external cloud
    • best practice
  • workflow management
    • rose cylc on jasmin
    • using cron
  • uncategorized
    • mobaxterm
    • requesting resources
    • processing requests for resources
    • acknowledging jasmin
    • approving requests for access
    • working with many linux groups
    • jasmin conditions of use
  1.   Batch Computing
  1. Home
  2. Docs
  3. Batch Computing
  4. Slurm queues

Slurm queues

 

Slurm   Queue   Partition   Sinfo   Lotus   Orchid  
Slurm   Queue   Partition   Sinfo   Lotus   Orchid  
Share via
JASMIN Help Site
Link copied to clipboard

Slurm queues/partitions for batch job submissions to the LOTUS & ORCHID clusters

On this page
Queue names   Queue details   State of queues   ‘sinfo’ Output field description:   How to choose a Slurm queue/partition   Test queue   Serial queues   Parallel queues  
 
Not yet reviewed for compatibility with the new cluster, April 2025. Please adapt using new submission instructions. This alert will be removed once updated.

Queue names  

The Slurm queues in the LOTUS cluster are:

  • test
  • short-serial
  • long-serial
  • par-single
  • par-multi
  • high-mem
  • short-serial-4hr

Each queue is has attributes of run-length limits (e.g. short, long) and resources. A full breakdown of each queue and its associated resources is shown below in Table 1.

Queue details  

Queues represent a set of pending jobs, lined up in a defined order, and waiting for their opportunity to use resources. The queue is specified in the job script file using Slurm scheduler directive like this:

#SBATCH -p <partition=queue_name>`

where <queue_name> is the name of the queue/partition (Table 1. column 1)

Table 1 summarises important specifications for each queue such as run time limits and the number of CPU core limits. If the queue is not specified, Slurm will schedule the job to the queue short-serial by default.

Table 1. LOTUS/Slurm queues and their specifications

Queue name Max run time Default run time Max CPU cores per job MaxCpuPerUserLimit Priority
test 4 hrs 1hr 8 8 30
short-serial 24 hrs 1hr 1 2000 30
par-single 48 hrs 1hr 16 300 25
par-multi 48 hrs 1hr 256 300 20
long-serial 168 hrs 1hr 1 300 10
high-mem 48 hrs 1hr 1 75 30
short-serial-4hr
(Note 3)
4 hrs 1hr 1 1000 30

Note 1 : Resources requested by a job must be within the resource allocation limits of the selected queue.

Note 2: The default value for --time=[hh:mm:ss] (predicted maximum wall time) is 1 hour for the all queues. If you do not specify this option and/or your job exceeds the default maximum run time limit then it will be terminated by the Slurm scheduler.

Note 3 : A user must specify the Slurm job account --account=short4hr when submitting a batch job to the short-serial-4hr queue.

State of queues  

The Slurm command sinfo reports the state of queues and nodes managed by Slurm. It has a wide variety of filtering, sorting, and formatting options.

sinfo
PARTITION     AVAIL  TIMELIMIT  NODES  STATE NODELIST
test             up    4:00:00     48   idle host[146-193]
short-serial*    up 1-00:00:00     48   idle host[146-193]
long-serial      up 7-00:00:00     48   idle host[146-193]
par-single       up 2-00:00:00     48   idle host[146-193]
par-multi        up 2-00:00:00     48   idle host[146-193]
high-mem         up 2-00:00:00     48   idle host[146-193]
lotus_gpu        up 7-00:00:00     48   idle host[146-193]
copy             up 7-00:00:00     48   idle host[146-193]
cpom-comet       up 7-00:00:00     48   idle host[146-193]
...
 
Queues other than the standard queues test , short-serial , long-serial, par-single, par-multi and high-mem should be ignored as they implement different job scheduling and control policies.

‘sinfo’ Output field description:  

By default, the Slurm command ‘sinfo’ displays the following information:

  • PARTITION : Partition name followed by * for the default queue/partition
  • AVAIL : State/availability of a queue/partition. Partition state: up or down.
  • TIMELIMIT : The maximum run time limit per job in each queue/partition is shown in TIMELIMIT in days- hours:minutes :seconds . e.g. 2-00:00:00 is two days maximum runtime limit
  • NODES : Count of nodes with this particular configuration e.g. 48 nodes
  • STATE : State of the nodes. Possible states include: allocated, down, drained, and idle. For example, the state “idle” means that the node is not allocated to any jobs and is available for use.
  • NODELIST List of node names associated with this queue/partition

The sinfo example below, reports more complete information about the partition/queue short-serial

sinfo --long --partition=short-serial
Tue May 12 18:04:54 2020
PARTITION    AVAIL TIMELIMIT JOB_SIZE  ROOT  OVERSUBS  GROUPS NODES    STATE NODELIST
short-serial* up  1-00:00:00  1-infinite  no  NO    all     48  idle host[146-193]

How to choose a Slurm queue/partition  

Test queue  

The test queue can be used to test new workflows and also to help new users to familiarise themselves with the Slurm batch system. Both serial and parallel code can be tested on the testqueue. The maximum runtime is 4 hrs and the maximum number of jobs per user is 8 job slots. The maximum number of cores for a parallel job e.g. MPI, OpenMP, or multi-threads is limited to 8 cores. The testqueue should be used when unsure of the job resource requirements and behavior at runtime because it has a confined set of LOTUS nodes (Intel node type) not shared with the other standard LOTUS queues.

Serial queues  

Serial and array jobs with a single CPU core should be submitted to one of the following serial queues depending on the job duration and the memory requirement. The default queue is short-serial

short-serial  

Serial and/or array jobs with a single CPU core each and run time less than 24 hrs should be submitted to the short-serial queue . This queue has the highest priority of 30. The maximum number of jobs that can be scheduled to start running from the short-serial is 2000 jobs whilst both job’s resources are available and user’ priority is high

long-serial  

Serial or array jobs with a single CPU core and run time greater than 24 hrs and less than 168 hrs (7 days) should be submitted to the queue long-serial . This queue has the lowest priority of 10 and hence jobs might take longer to be scheduled to run relatively to other jobs in higher priority queues.

high-mem  

Serial or array jobs with a single CPU core and high memory requirement (> 64 GB) should be submitted to the high-mem queue and the required memory must be specified --mem=XXX (XXX is in MB units). The job should not exceed the maximum run time limit of 48hrs. This queue is not configured to accept exclusive jobs.

Parallel queues  

Jobs requiring more than one CPU core should be submitted to one of the following parallel queues depending on the type of parallelisms such as shared memory or distributed memory jobs.

par-single  

Shared memory multi-threaded jobs with a maximum of 16 threads should be submitted to the par-single queue . Each thread should be allocated one CPU core. Oversubscribing the number of threads to the CPU cores will cause the job to run very slow. The number of CPU cores should be specified via the submission command line sbatch -n <number of CPU cores> or by adding the Slurm directive #SBATCH -n <number of CPU cores>in the job script file. An example is shown below:

sbatch --ntasks=4 --partition=par-single < myjobscript

Note: Jobs submitted with a number of CPU cores greater than 16 will be terminated (killed) by the Slurm scheduler with the following statement in the job output file:

par-multi  

Distributed memory jobs with inter-node communication using the MPI library should be submitted to the par-multi queue . A single MPI process (rank) should be allocated a single CPU core. The number of CPU cores should be specified via the Slurm submission command flag sbatch -n <number of CPU cores> or by adding the Slurm directive #SBATCH -n <number of CPU cores> to the job script file. An example is shown below:

sbatch --ntasks=4 --partition=par-multi < myjobscript

Note 1: The number of CPU cores gets passed from Slurm submission flag -n . Do not add the -np flag to mpirun command .

Note 2: Slurm will reject a job that requires a number of CPU cores greater than the limit of 256.

Last updated on 2025-04-08 as part of:  Move descriptions out of article body (71ffd6e71)
On this page:
Queue names   Queue details   State of queues   ‘sinfo’ Output field description:   How to choose a Slurm queue/partition   Test queue   Serial queues   Parallel queues  
Follow us

Social media & development

   

Useful links

  • CEDA Archive 
  • CEDA Catalogue 
  • JASMIN 
  • JASMIN Accounts Portal 
  • JASMIN Projects Portal 
  • JASMIN Cloud Portal 
  • JASMIN Notebooks Service 
  • JASMIN Community Discussions 

Contact us

  • Helpdesk
UKRI/STFC logo
UKRI/NERC logo
NCAS logo
NCEO logo
Accessibility | Terms and Conditions | Privacy and Cookies
Copyright © 2025 Science and Technology Facilities Council.
Hinode theme for Hugo licensed under Creative Commons (CC BY-NC-SA 4.0).
JASMIN Help Site
Code copied to clipboard