Jaspy Software Environments (Python 3, R and other tools)

This page provides details of the "Jaspy" software environments that provide access to Python 3, R and a range of other tools on JASMIN. The article includes:

Overview

Jaspy is a toolkit for managing and deploying Conda environments that include Python, R and other packages. Jaspy is used to provide software environments of common packages on the scientific analysis servers and LOTUS cluster on JASMIN.
One advantage of Jaspy is that multiple environments can co-exist on the same platform. This allows us to retain previous environments and provide new ones simultaneously. This may be particularly useful for scientists undertaking long-running studies that require a consistent software environment to ensure reproducibility and continuity.

Working with Jaspy environments

Quickstart for Python 3 environment

If you want to get on, you can select a Jaspy environment to "activate". This means that once you have run these commands then the various tools and libraries will be available in your current session.
module load jaspy

Activating the environment in scripts

If you want a particular script to activate a Jaspy environment then add the "module" command to it, e.g.:

#!/bin/bash
module load jaspy
python do-something.py

Setting your profile to always use a Jaspy environment

If you want all your JASMIN sessions to use a particular Jaspy environment then you can add the "module load jaspy" command to your ".bashrc" file at: $HOME/.bashrc . In order to avoid issues with using "module load" on unsupported servers, please wrap the call in an "if" clause, such as:

if [[ $(hostname) =~ (sci[0-9]|host[0-9]|cylc) ]] ; then 
    module load jaspy
fi

Discover which environments are available

You can list the currently available Jaspy environments using:
$ module avail jaspy

-----------  /apps/modulefiles  -------------
   jaspy/2.7/r20190715    jaspy/3.7/r20190627    jaspy/3.8/r20211105        
   jaspy/3.7/r20181219    jaspy/3.7/r20200606    jaspy/3.10/r20220721 (D)
   jaspy/3.7/r20190612    jaspy/3.7/r20210320    jaspy/3.10/r20230718 (L)

  Where:
   L:  Module is loaded
   D:  Default Module

This lists all jaspy modules (i.e. environments) that can be loaded.

Jaspy Python 3.7+ (plus other tools) environments

The packages available in the Jaspy environments can be found by searching the GitHub repository where the Conda environment files are defined. This table lists all the Jaspy Python 3.7+ environments provided on JASMIN and specifies the current (default) version.
Jaspy Python 3.7 Environment Versioned list of software packages Default? Comments / Issues
jaspy/3.10/r20230718 List of packages including versions
Not yet (to be scheduled soon)
jaspy/3.10/r20220721 List of packages including versions Yes (from: 18/10/2022) NCO and NCL have now been moved to the "jasmin-sci" packages installation.
jaspy/3.8/r20211105 List of packages including versions No (was default: 16/11/2021 - 17/10/2022) Known problem with NCL rendering Shapefiles (see issue). Some packages were removed in this release due to dependency problems: theano, pymc3, pystan, pyngl, pyferret (see issue).
jaspy/3.7/r20210320 List of packages including versions
No (was default:
20/05/2021 - 16/11/2021)
Known problem with NCL rendering Shapefiles (see issue)
jaspy/3.7/r20200606 List of packages including versions No
jaspy/3.7/r20181219
List of packages including versions No

Jaspy Python 2.7 (plus other tools) environments

This table lists all the Jaspy Python 2.7 environments provided on JASMIN and specifies the current (default) version.

Jaspy Python 2.7 Environment   Versioned list of software packages Default?
jaspy/2.7/r20190715   List of packages including versions
Yes
   

Jaspy R environments (Jasr)

Environments for the "R" programming language are packaged into separate software environments, known as "Jasr". This table lists all the Jaspy R environments provided on JASMIN and specifies the current (default) version.

Jaspy R Environment ("Jasr") Versioned list of software packages Default?
jasr/4.2/r20230718/ List of packages including versions
Not yet (to be scheduled soon)
jasr/4.0/r20220729 List of packages including versions
Yes (from: 18/10/2022)
jasr/4.0/r20211110   List of packages including versions No
(was default: 16/11/2021 - 17/10/2022)
   

The available R environments can be listed with:

module avail jasr
	

Understanding versioning with Jaspy/Jasr

Jaspy environments are labelled as "jaspy/<python_version>/<release>". The environment is selected and activated using the "module load" command:
module load jaspy/3.7/r20210320
However, if you wish to get the latest environment for a given Python version you can omit the "<release>", as follows:
module load jaspy/3.7
	
And if you just want the most up-to-date Python you can even omit the "<python_version>", as follows:
module load jaspy
	
Please note: if you choose to omit the "<release>" and "<python_version>" components then it is important to be aware that the resulting environment may differ over time. For continuity, you may wish to specify the full environment specification.

How Jaspy works: managing Python and non-Python packages using conda

Jaspy is a framework for managing multiple Python (and other) environments simultaneously on a single platform. It was created in order to meet the requirements tabulated below.

Requirement Details Jaspy solution Further info
Reproducibility
  1. Generate a specific set of packages and versions from a generic set of requirements.
  1. Conda has a powerful package-management workflow:
    1. Begin with a minimal set of package/version requirements.
    2. Generate a consistent environment.
    3. Provide a detailed description of all exact packages/versions in the environment.
Conda: https://docs.conda.io
jaspy-manager: https://github.com/cedadev/jaspy-manager/blob/master/README.md
CEDA jaspy environments: https://github.com/cedadev/ceda-jaspy-envs
Documentation Provide an appropriate level of documentation detailing which software packages exist in each release. We use Conda "environment files" to build the environments. These list the packages and versions and are stored in public GitHub repositories, so each environment is documented as a collection of packages/versions. See: https://github.com/cedadev/jaspy-manager/blob/master/README.md
Example package list: https://github.com/cedadev/ceda-jaspy-envs/blob/master/environments/py3.7/m3-4.5.11/jaspy3.7-m3-4.5.11-r20181219/packages.txt
Multiple simultaneous environments Allow multiple, but separate, software environments to co-exist on a single operating system. Conda is designed to allow multiple environments to co-exist. Within jaspy it is possible to document each environment. Therefore, multiple environments can be deployed on one system. Key advantages are:
  • Supporting multiple versions of Python and side-by-side.
  • Releasing an update to an environment as a "pre-release" so that users can adapt their code and test it whilst still having access to the "current" (production) environment.
Manageability Provide tools to easily construct, test, deploy, document and reproduce software environments. Jaspy builds upon a set of excellent Conda command-line tools that simplify the package management process. Jaspy wraps the Conda functionality so that command-line tools can be used to build, test, deploy and distribute Conda environments for use by our community.

Updates and tracking of Jaspy/Jasr environments

History of environments on JASMIN

Please see the "Jaspy Python 3.7+ (and other tools) environments" section above for information about releases on JASMIN.

Which environment is "current"?

Please refer to the "Jaspy Python 3.7+ (and other tools) environments" section above for information about the current release on JASMIN.

Citing Jaspy environments

Can I cite a jaspy (conda) environment?

We do not yet have an agreed approach for citing a Jaspy environment. However, you can refer to the environment description URLs given in the table above. These provide a definitive list of the software packages, their versions and other information.

Requesting updates to a Jaspy environment

If you would like us to add a new package, or an updated version, to the Jaspy environments on JASMIN then please use one of the following approaches:
  1. E-mail the JASMIN Helpdesk with the subject: "Request for Jaspy update: <package name>"
  2. Get a GitHub account and add an issue to the `ceda-jaspy-envs` repository at:
    1. https://github.com/cedadev/ceda-jaspy-envs/issues/new

Conda method of "activating" Jaspy environments

Jaspy environments can also be activated in a more traditional way using standard the standard conda approach, for example:
$ export PATH=/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14//bin/conda:$PATH
$ source activate
$ conda activate jaspy3.7-m3-4.6.14-r20210320
	
This has the same result as the "module load" approach. The naming of the environment identifiers includes the "Miniconda" version used to generate the environment. The "module load" approach is recommended as the standard method for activating Jaspy environments.

Using Jaspy beyond JASMIN

Jaspy is a versatile and generic tool for managing multiple conda environments. The code is open source, and more information is available at:

https://github.com/cedadev/jaspy-manager

Still need help? Contact Us Contact Us