Jaspy Software Environments (Python 3, R and other tools)
This page provides details of the "Jaspy" software environments that provide access to Python 3, R and a range of other tools on JASMIN. The article includes:
- Working with Jaspy environments
- Discovering which environments are available
- Jaspy Python (and other tools) environments
- Jaspy R environments (Jasr)
- How Jaspy works
- Changes to software environments
Overview
Jaspy is a toolkit for managing and deploying Conda environments that include Python, R and other packages. Jaspy is used to provide software environments of common packages on the scientific analysis servers and LOTUS cluster on JASMIN.
One advantage of Jaspy is that multiple environments can co-exist on the same platform. This allows us to retain previous environments and provide new ones simultaneously. This may be particularly useful for scientists undertaking long-running studies that require a consistent software environment to ensure reproducibility and continuity.
Working with Jaspy environments
Quickstart for Python 3 environment
If you want to get on, you can select a Jaspy environment to "activate". This means that once you have run these commands then the various tools and libraries will be available in your current session.
module load jaspy
Activating the environment in scripts
If you want a particular script to activate a Jaspy environment then add the "module" command to it, e.g.:
#!/bin/bash module load jaspy python do-something.py
Setting your profile to always use a Jaspy environment
If you want all your JASMIN sessions to use a particular Jaspy environment then you can add the "module load jaspy" command to your ".bashrc" file at: $HOME/.bashrc . In order to avoid issues with using "module load" on unsupported servers, please wrap the call in an "if" clause, such as:
if [[ $(hostname) =~ (sci[0-9]|host[0-9]|cylc) ]] ; then
module load jaspy
fi
Discover which environments are available
You can list the currently available Jaspy environments using:
$ module avail jaspy ----------- /apps/modulefiles ------------- jaspy/2.7/r20190715 jaspy/3.7/r20190627 jaspy/3.8/r20211105 jaspy/3.7/r20181219 jaspy/3.7/r20200606 jaspy/3.10/r20220721 (D) jaspy/3.7/r20190612 jaspy/3.7/r20210320 jaspy/3.10/r20230718 (L) Where: L: Module is loaded D: Default Module
This lists all jaspy modules (i.e. environments) that can be loaded.
Jaspy Python 3.7+ (plus other tools) environments
The packages available in the Jaspy environments can be found by searching the GitHub repository where the Conda environment files are defined. This table lists all the Jaspy Python 3.7+ environments provided on JASMIN and specifies the current (default) version.
| Jaspy Python 3.7 Environment | Versioned list of software packages | Default? | Comments / Issues | |
| jaspy/3.10/r20230718 | List of packages including versions |
Not yet (to be scheduled soon) | ||
| jaspy/3.10/r20220721 | List of packages including versions | Yes (from: 18/10/2022) | NCO and NCL have now been moved to the "jasmin-sci" packages installation. | |
| jaspy/3.8/r20211105 | List of packages including versions | No (was default: 16/11/2021 - 17/10/2022) | Known problem with NCL rendering Shapefiles (see issue). Some packages were removed in this release due to dependency problems: theano, pymc3, pystan, pyngl, pyferret (see issue). | |
| jaspy/3.7/r20210320 | List of packages including versions |
No (was default: 20/05/2021 - 16/11/2021) |
Known problem with NCL rendering Shapefiles (see issue) | |
| jaspy/3.7/r20200606 | List of packages including versions | No | ||
| jaspy/3.7/r20181219 |
List of packages including versions | No |
Jaspy Python 2.7 (plus other tools) environments
This table lists all the Jaspy Python 2.7 environments provided on JASMIN and specifies the current (default) version.
| Jaspy Python 2.7 Environment | Versioned list of software packages | Default? | |
| jaspy/2.7/r20190715 | List of packages including versions |
Yes | |
Jaspy R environments (Jasr)
Environments for the "R" programming language are packaged into separate software environments, known as "Jasr". This table lists all the Jaspy R environments provided on JASMIN and specifies the current (default) version.
| Jaspy R Environment ("Jasr") | Versioned list of software packages | Default? | |
| jasr/4.2/r20230718/ | List of packages including versions |
Not yet (to be scheduled soon) | |
| jasr/4.0/r20220729 | List of packages including versions |
Yes (from: 18/10/2022) |
|
| jasr/4.0/r20211110 | List of packages including versions | No (was default: 16/11/2021 - 17/10/2022) |
|
The available R environments can be listed with:
module avail jasr
Understanding versioning with Jaspy/Jasr
Jaspy environments are labelled as "jaspy/<python_version>/<release>". The environment is selected and activated using the "module load" command:
module load jaspy/3.7/r20210320
However, if you wish to get the latest environment for a given Python version you can omit the "<release>", as follows:
module load jaspy/3.7
And if you just want the most up-to-date Python you can even omit the "<python_version>", as follows:
module load jaspy
Please note: if you choose to omit the "<release>" and "<python_version>" components then it is important to be aware that the resulting environment may differ over time. For continuity, you may wish to specify the full environment specification.
How Jaspy works: managing Python and non-Python packages using conda
Jaspy is a framework for managing multiple Python (and other) environments simultaneously on a single platform. It was created in order to meet the requirements tabulated below.
| Requirement | Details | Jaspy solution | Further info | ||
| Reproducibility |
|
|
Conda: https://docs.conda.io jaspy-manager: https://github.com/cedadev/jaspy-manager/blob/master/README.md CEDA jaspy environments: https://github.com/cedadev/ceda-jaspy-envs |
||
| Documentation | Provide an appropriate level of documentation detailing which software packages exist in each release. | We use Conda "environment files" to build the environments. These list the packages and versions and are stored in public GitHub repositories, so each environment is documented as a collection of packages/versions. | See: https://github.com/cedadev/jaspy-manager/blob/master/README.md Example package list: https://github.com/cedadev/ceda-jaspy-envs/blob/master/environments/py3.7/m3-4.5.11/jaspy3.7-m3-4.5.11-r20181219/packages.txt |
||
| Multiple simultaneous environments | Allow multiple, but separate, software environments to co-exist on a single operating system. | Conda is designed to allow multiple environments to co-exist. Within jaspy it is possible to document each environment. Therefore, multiple environments can be deployed on one system. Key advantages are:
|
|||
| Manageability | Provide tools to easily construct, test, deploy, document and reproduce software environments. | Jaspy builds upon a set of excellent Conda command-line tools that simplify the package management process. Jaspy wraps the Conda functionality so that command-line tools can be used to build, test, deploy and distribute Conda environments for use by our community. |
Updates and tracking of Jaspy/Jasr environments
History of environments on JASMIN
Please see the "Jaspy Python 3.7+ (and other tools) environments" section above for information about releases on JASMIN.
Which environment is "current"?
Please refer to the "Jaspy Python 3.7+ (and other tools) environments" section above for information about the current release on JASMIN.
Citing Jaspy environments
Can I cite a jaspy (conda) environment?
We do not yet have an agreed approach for citing a Jaspy environment. However, you can refer to the environment description URLs given in the table above. These provide a definitive list of the software packages, their versions and other information.
Requesting updates to a Jaspy environment
If you would like us to add a new package, or an updated version, to the Jaspy environments on JASMIN then please use one of the following approaches:
- E-mail the JASMIN Helpdesk with the subject: "Request for Jaspy update: <package name>"
- Get a GitHub account and add an issue to the `ceda-jaspy-envs` repository at:
Conda method of "activating" Jaspy environments
Jaspy environments can also be activated in a more traditional way using standard the standard conda approach, for example:
$ export PATH=/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14//bin/conda:$PATH $ source activate $ conda activate jaspy3.7-m3-4.6.14-r20210320
This has the same result as the "module load" approach. The naming of the environment identifiers includes the "Miniconda" version used to generate the environment. The "module load" approach is recommended as the standard method for activating Jaspy environments.
Using Jaspy beyond JASMIN
Jaspy is a versatile and generic tool for managing multiple conda environments. The code is open source, and more information is available at: