Jaspy Software Environments (Python 3 and other tools, on RHEL6/7)

This page provides details of the "Jaspy" software environments that provide access to Python 3 and a range of other tools on JASMIN. The article includes:

  • Working with Jaspy environments
  • Discovering which environments are available
  • How Jaspy works
  • Changes to software environments

Overview

Jaspy is a toolkit for managing and deploying Conda environments that include both python and non-python packages. Jaspy environments are due to replace the JAP on the RedHat Enterprise Linux (RHEL) 6 and 7 scientific analysis servers and LOTUS cluster on JASMIN.
One advantage of Jaspy is that multiple environments can co-exist on the same platform. We can therefore retain previous environments and provide simultaneously. This may be particularly useful for scientists undertaking long-running studies that require a consistent software environment to ensure reproducibility and continuity.

Working with Jaspy environments

Quick start for Python 3 environment

If you want to get on, you can select a Jaspy environment to "activate". This means that once you have run these commands then the various tools and libraries will be available in your current session.
module load jaspy

Activating the environment in scripts

If you want a particular script to activate a Jaspy environment then add the "module" command to it, e.g.:

#!/bin/bash
module load jaspy
python do-something.py

Setting your profile to always use a Jaspy environment

If you want all your JASMIN sessions to use a particular Jaspy environment then you can add the "module load jaspy" command to your ".bashrc" file at: $HOME/.bashrc . In order to avoid issues with using "module load" on unsupported servers, please wrap the call in an "if" clause, such as:

if [[ $(hostname) =~ (-sci[0-9]|host[0-9]|jasmin-cylc) ]] ; then 
    module load jaspy
fi

Discover which environments are available

You can list the currently available Jaspy environments using:
$ module avail jaspy
------------------------/apps/modulefiles ----------------------------
jaspy/2.7/r20190715    jaspy/3.7/r20181219

This lists any jaspy modules (i.e. environments) that can be loaded.

Jaspy Python 3.7 (plus other tools)

The packages available in the Jaspy environments can be found by searching the GitHub repository where the Conda environment files are defined. This table lists all the Jaspy Python 3.7 environments provided on JASMIN:
Jaspy Python 3.7 Environment List of software packages Versioned list of software packages
jaspy/3.7/r20181219 https://github.com/cedadev/ceda-jaspy-envs/blob/master/environments/py3.7/m3-4.5.11/jaspy3.7-m3-4.5.11-r20181219/initial.yml https://github.com/cedadev/ceda-jaspy-envs/blob/master/environments/py3.7/m3-4.5.11/jaspy3.7-m3-4.5.11-r20181219/packages.txt

Jaspy Python 2.7 (plus other tools)

This table lists all the Jaspy Python 2.7 environments provided on JASMIN:

Jaspy Python 2.7 Environment   List of software packages Versioned list of software packages
jaspy/2.7/r20190715     https://github.com/cedadev/ceda-jaspy-envs/blob/master/environments/py2.7/m2-4.6.14/jaspy2.7-m2-4.6.14-r20190715/initial.yml
https://github.com/cedadev/ceda-jaspy-envs/blob/master/environments/py2.7/m2-4.6.14/jaspy2.7-m2-4.6.14-r20190715/packages.txt
   
Please also see the previous Python 2.7 environment details as provided via the  JASMIN Analysis Platform.

Understanding Jaspy versioning

Jaspy environments are labelled as "jaspy/<python_version>/<release>". The environment is selected and activated using the "module load" command:
module load jaspy/3.7/r20181219
However, if you wish to get the latest environment for a given Python version you can omit the "<release>", as follows:
module load jaspy/3.7
	
And if you just want the most up-to-date Python you can even omit the "<python_version>", as follows:
module load jaspy
	
Please note: if you choose to omit the "<release>" and "<python_version>" components then it is important to be aware that the resulting environment may differ over time. For continuity you may wish to specify the full environment specification.

How Jaspy works: managing Python and non-Python packages using conda

Jaspy is a framework for managing multiple Python (and other) environments simultaneously on a single platform. It was created in order to meet requirements tabulated below.

Requirement Details Jaspy solution Further info
Reproducibility
  1. Generate a specific set of packages and versions from a generic set of requirements.
  2. Maintain access to all package binaries over time so that two equivalent installations result in exactly the same software versions.
  1. Conda has a powerful package-management workflow:
    1. Begin with a minimal set of package/version requirements.
    2. Generate a consistent environment.
    3. Provide a detailed description of all exact packages/versions in the environment.
  2. Even though Conda channels may change over time, Jaspy saves a copy of the binaries to a local, backed-up server and indexes them in a local channel. This means they remain available even if the original package/channel is removed.
Conda: https://docs.conda.io
jaspy-manager: https://github.com/cedadev/jaspy-manager/blob/master/README.md
CEDA jaspy environments: https://github.com/cedadev/ceda-jaspy-envs
Documentation Provide an appropriate level of documentation detailing which software packages exist in each release. We use Conda "environment files" to build the environments. These list the packages and versions and are stored in public GitHub repositories so each environment is documented as a collection of packages/versions. See: https://github.com/cedadev/jaspy-manager/blob/master/README.md
Example package list: https://github.com/cedadev/ceda-jaspy-envs/blob/master/environments/py3.7/m3-4.5.11/jaspy3.7-m3-4.5.11-r20181219/packages.txt
Multiple simultaneous environments Allow multiple, but separate, software environments to co-exist on a single operating system. Conda is designed to allow multiple environments to co-exist. Within jaspy it is possible to document each environment. Therefore, multiple environments can be deployed on one system. Key advantages are:
  • Supporting Python 2 and Python 3 side-by-side.
  • Releasing an update to an environment as a "pre-release" so that users can adapt their code and test it whilst still have access to the "current" (production) environment.
Manageability Provide tools to easily construct, test, deploy, document and reproduce software environments. Jaspy builds upon a set of excellent Conda command-line tools that simplify the package management process. Jaspy wraps the Conda functionality so that command-line tools can be used to build, test, deploy and distribute Conda environments for use by our community.

Updates and tracking of Jaspy environments

History of Jaspy environments on JASMIN

Since Jaspy is relatively new please refer to the "Jaspy Python 3.7 (and other tools)" section above for information about releases on JASMIN.

Which Jaspy environment is "current"?

Since Jaspy is relatively new please refer to the "Jaspy Python 3.7 (and other tools)" section above for information about the current release on JASMIN.

Citing Jaspy environments

Can I cite a jaspy (conda) environment?

We do not yet have an agreed approach for citing a Jaspy environment. However, you can refer to the environment description URLs given in the table above. These provide a definitive list of the software packages, their versions and other information.

Requesting updates to a Jaspy environment

If you would like us to add a new package, or an updated version, to the Jaspy environments on JASMIN then please use one of the following approaches:
  1. E-mail the CEDA Helpdesk with the subject: "Request for Jaspy update: <package name>"
  2. Get a GitHub account and add an issue to the `ceda-jaspy-envs` repository at:
    1. https://github.com/cedadev/ceda-jaspy-envs/issues/new

Alternative method of "activating" Jaspy environments

Jaspy environments can also be activated in a more traditional way using standard the standard conda approach, for example:
$ export PATH=/apps/contrib/jaspy/miniconda_envs/jaspy3.7/m3-4.5.11/bin:$PATH
$ source activate jaspy3.7-m3-4.5.11-r20181219
	
This has the same result as the "module load" approach. The naming of the environment identifiers includes the "Miniconda" version used to generate the environment. The "module load" approach is recommended as the standard method for activating Jaspy environments.

Using Jaspy outside of JASMIN

Jaspy is a versatile and generic tool for managing multiple conda environments. The code is open source and more information is available at:

https://github.com/cedadev/jaspy-manager

Still need help? Contact Us Contact Us