JASMIN software changes: migration to CentOS7 (2020)
JASMIN software changes: migration to CentOS7 (2020)
This article explains the changes in the provision of common software on JASMIN (“sci” servers and LOTUS) when upgrading to main operating system from RedHat Enterprise Linux 6 (RHEL6) to CentOS7. It answers for the following questions:
The upgrade from RHEL6 to introduces a new way of working with software environments on JASMIN.
The previous system involved installations via RPM using a collection of packages known as the “JASMIN Analysis Platform” (JAP).
The current system uses two approaches to providing software environments and packages:
This change applies to all generally available JASMIN servers and the LOTUS cluster. These include:
The reason for moving away from the JAP was to provide a system that could support multiple versions of software packages, and environments, on a single platform. From the perspective of reproducible science, the Jaspy approach is more useful because it:
From a management point-of-view, Jaspy builds on the packaging tool “ conda ” and the community repositories known as “ conda-forge ”. These tools are widely used in the scientific community and provide many components that are re-used in Jaspy.
Most packages that were previously provided as part of the JASMIN Analysis Platform (JAP) are now included in Jaspy environments or the “jasmin-sci” environment. Some packages have however been dropped. The treatment of certain packages is documented on the “ extra-sci- packages ” GitHub repository. A full table of packages that were in JAP and their mappings to Jaspy/“jasmin- sci” is provided below.
If you want to access most of the packages you can find them in the most recent Jaspy environment. This can be activated using:
module load jaspyIf you need packages that are provided in the “jasmin-sci” environment then you can activate it using:
module load jasmin-sciHere is a full table of packages that were provided via JAP, and how they are provided for use on CentOS7 machines (as of initial release).
In the summary column:
| NAME (in JAP) | Version in Jaspy 3.7 (conda) | Version in Jaspy 2.7 (conda) | Version in RPMs (base or jasmin-sci SCL) | Summary: (C)onda/ (R)PM / (N)one | Comments |
|---|---|---|---|---|---|
| arpack | 3.6.3 | 3.6.3 | C | ||
| atlas | 3.8.4 | 3.10.1 | R | ||
| atlas-devel | 3.8.4 | 3.10.1 | R | ||
| bbcp | N | deprecated | |||
| bbcp-config | N | deprecated | |||
| blas | 1.1 | 1.0 | C | ||
| boost-devel | 1.70.0 | C | |||
| cdo | 1.9.5 | C | |||
| cmip6-cmor-tables | N | groups advised to maintain own CMOR | |||
| cmor-libs | N | groups advised to maintain own CMOR | |||
| coda | 0.19_1 | C | |||
| ddd | 3.3.12 | R | |||
| diffuse | 0.4.8 | R | |||
| dvipng | 1.14 | R | package texlive-dvipng-svn26689 | ||
| eccodes | 2.9.2 | C | |||
| eccodes-devel | 2.9.2 | C | |||
| eccodes-fortran | 2.9.2 | C | |||
| eccodes-python27 | 2.9.2 | C | |||
| emacs | 26.1 | 24.3 | R | emacs is broken in jaspy 2.7 - | |
| /usr/bin/emacs may be required to override | |||||
| emacs-common-ess | N | might provide later | |||
| emacs-ess | N | might provide later | |||
| emacs-ess-el | N | might provide later | |||
| emacs-gnuplot | 4.6.2 | R | |||
| emos | N | deprecated | |||
| esmf | 7.1.0r | 7.1.0r | C | ||
| esmf-doc | N | See here and ESMF_refdoc | |||
| esmf-python27 | 7.1.0r | 7.1.0r | C | ||
| ferret | 7.5.0 | R | |||
| fftw | 3.3.8 | 3.3.8 | C | ||
| fftw-devel | 3.3.8 | 3.3.8 | C | ||
| firefox | N | ||||
| flex-devel | 2.6.4 | C | |||
| gcc-gfortran | 4.8.5 | R | |||
| gdal | 2.2.4 | 2.0.0 | C | ||
| gdal-devel | N | gdal provided as “gdal” | |||
| gdal-doc | N | gdal provided as “gdal” | |||
| gdal-java | N | ||||
| gdal-javadoc | N | ||||
| gdal-libs | N | ||||
| gdal-perl | N | ||||
| gdal-python27 | 2.2.4 | 2.0.0 | C | ||
| geany | 1.31 | R | |||
| geos | 3.6.2 | 3.7.1 | C | ||
| geos-devel | 3.6.2 | 3.7.1 | 3.4.2 | R | |
| git | 2.20.1 | 2.20.1 | C | ||
| gitk | 1.8.3.1 | R | |||
| glibc-static | 2.17 | R | |||
| gnuplot | 4.6.2 | R | |||
| grads | 2.0.2 | R | |||
| GraphicsMagick-c++ | 1.3.32 | R | |||
| graphviz | 2.38.0 | 2.38.0 | C | ||
| graphviz-gd | N | ||||
| graphviz-python27 | 2.38.0 | 2.38.0 | C | ||
| grass | 6.4.4 | R | |||
| grass-devel | 6.4.4 | R | |||
| grass-libs | 6.4.4 | R | |||
| grib_api | N | deprecated - use eccodes | |||
| grib_api-devel | N | deprecated - use eccodes | |||
| grib_api-fortran | N | deprecated - use eccodes | |||
| grib_api-python27 | N | deprecated - use eccodes | |||
| gsl | 2.2.1 | 2.5 | C | ||
| gsl-devel | 2.2.1 | 2.5 | C | ||
| gsl-static | 2.2.1 | 2.5 | C | ||
| gtk2 | 2.24.31 | C | |||
| gtk2-devel | 2.24.31 | C | |||
| gv | 3.7.4 | R | |||
| hdf | 4.2.13 | 4.2.13 | C | ||
| hdf-devel | 4.2.13 | 4.2.13 | C | ||
| hdf5 | 1.10.3 | 1.10.1 | C | ||
| hdf5-devel | 1.10.3 | 1.10.1 | C | ||
| hdfeos2 | 2.2 | 2.20 | 20.1.00 | CR | |
| hdfeos5 | 5.1.16 | 5.1.16 | C | ||
| ImageMagick | 7.0.8_16 | 7.0.8_10 | C | ||
| JAGS | N | available as “rjags” | |||
| jasper-devel | 1.900.1 | 2.0.14 | C | ||
| ksh | 20120801 | R | |||
| lapack | 3.6.1 | 3.6.1 | C | ||
| lapack-devel | 3.6.1 | 3.6.1 | C | ||
| leafpad | 0.8.18 | R | |||
| libcdms | 3.1.0 | 3.0.1 | C | ||
| libcrayutil | 20121128 | R | |||
| libcurl-devel | 7.29.0 | R | |||
| libdrs | 3.1.0 | 3.1.0 | 3.1.2 | CR | |
| libuuid-devel | 2.32.1 | 2.32.1 | 2.23.2 | CR | |
| llvm-devel | 3.3 | C | |||
| lxterminal | 0.3.2 | R | |||
| mercurial | 4.9.1 | 2.6.2 | R | ||
| mo_unpack | 2.0.1 | R | |||
| mtk | 1.4.5 | R | |||
| mtk-devel | 1.4.5 | R | |||
| mtk-python27 | N | see build instructions for user | |||
| ncBrowse | N | deprecated | |||
| nccmp | 1.8.3.1 | R | |||
| ncl | 6.5.0 | C | |||
| nco | 4.7.8 | C | |||
| nco-devel | 4.7.8 | C | |||
| ncview | 2.1.2 | R | |||
| nedit | 5.7 | R | |||
| netcdf | 4.6.1 | 4.6.1 | 4.3.3.1 | CR | called libnetcdf in conda |
| netcdf-c++ | 4.2.1 | 4.3.0 | C | called netcdf-cxx4 in conda | |
| netcdf-c++-devel | 4.2.1 | 4.3.0 | C | called netcdf-cxx4 in conda | |
| netcdf-devel | 4.6.1 | 4.6.1 | 4.3.3.1 | CR | called libnetcdf in |
| conda | |||||
| netcdf-fortran | 4.4.4 | 4.4.4 | 4.2 | CR | |
| netcdf-fortran-devel | 4.4.4 | 4.4.4 | 4.2 | CR | |
| octave | 3.8.2 | R | |||
| octave-devel | 3.8.2 | R | |||
| octave-doc | N | docs are on https://octave.org/doc/ | |||
| octave-netcdf | 1.0.6 | R | |||
| octave-octcdf | N | deprecated | |||
| p7zip | 16.02 | R | |||
| parallel | 20190522 | C | |||
| pdftk | N | discontinued | |||
| perl-core | 5.16.3 | R | |||
| perl-devel | 5.16.3 | R | |||
| perl-Image-ExifTool | 11.7 | R | |||
| perl-XML-Parser | 2.44_01 | C | |||
| postgresql-devel | 9.2.24 | R | |||
| proj | 4.9.3 | 5.2.0 | 4.8.0 | CR | called proj4 in conda |
| proj-devel | 4.9.3 | 5.2.0 | 4.8.0 | CR | called proj4 in conda |
| proj-epsg | 4.9.3 | 5.2.0 | 4.8.0 | CR | called proj4 in conda |
| proj-nad | 4.9.3 | 5.2.0 | 4.8.0 | CR | called proj4 in conda |
| proj-static | 4.9.3 | 5.2.0 | 4.8.0 | CR | called proj4 in conda |
| python27 | 3.7.1 | 2.7.15 | C | ||
| python27-alabaster | 0.7.12 | 0.7.12 | C | ||
| python27-astral | 1.9.2 | C | |||
| python27-Babel | 2.6.0 | 2.7.0 | C | ||
| python27-backports-common | 1.0 | C | |||
| python27-backports-functools_lru_cache | 1.5 | C | |||
| python27-backports-ssl_match_hostname | 1.0 | C | |||
| python27-basemap | 1.2.0 | 1.2.0 | C | ||
| python27-biggus | 0.15.0 | C | |||
| python27-boto3 | 1.9.67 | 1.9.188 | C | ||
| python27-botocore | 1.12.68 | 1.12.188 | C | ||
| python27-cartopy | 0.17.0 | 0.16.0 | C | ||
| python27-ccplot | P | ||||
| python27-cdat_lite | N | deprecated | |||
| python27-cerbere | N | ||||
| python27-certifi | 2018.11.29 | C | |||
| python27-cf | 2.3.6 | C | called cf-python in conda | ||
| python27-cf-checker | 3.1.1 | C | |||
| python27-cf-plot | 2.4.10 | C | |||
| python27-cf-units | 2.0.2 | 2.1.1 | C | ||
| python27-cf-view | N | no longer supported | |||
| python27-cftime | 1.0.3.4 | 1.0.1 | C | ||
| python27-chardet | 3.0.4 | 3.0.4 | C | ||
| python27-cis | 1.6.0 | C | |||
| python27-cloudpickle | 0.6.1 | 1.2.1 | C | ||
| python27-cmor | N | groups advised to maintain own CMOR | |||
| python27-cycler | 0.10.0 | 0.10.0 | C | ||
| python27-Cython | 0.29.2 | 0.29.12 | C | ||
| python27-dask | 1.0.0 | 1.2.2 | C | ||
| python27-dateutil | C | As “dateutil” | |||
| python27-descartes | 1.1.0 | 1.1.0 | C | ||
| python27-docutils | 0.14 | 0.14 | C | ||
| python27-ecmwf-api-client | N | To be added soon. | |||
| python27-emcee | 2.2.1 | 2.2.1 | C | ||
| python27-enum34 | 1.1.6 | C | |||
| python27-eofs | 1.3.1 | 1.4.0 | C | ||
| python27-esgf-pyclient | 0.1.8 | C | |||
| python27-filelock | 3.0.10 | 3.0.10 | C | ||
| python27-Fiona | 1.7.13 | C | |||
| python27-geopandas | 0.4.0 | C | |||
| python27-h5py | 2.8.0 | 2.8.0 | C | ||
| python27-httplib2 | 0.13.0 | C | |||
| python27-idna | 2.8 | 2.8 | C | ||
| python27-ilamb | 2.3.1 | C | |||
| python27-ImageHash | 4.0 | 4.0 | C | ||
| python27-imagesize | 1.1.0 | 1.1.0 | C | ||
| python27-ipython | 7.2.0 | 5.8.0 | C | ||
| python27-iris_sample_data | 2.1.0 | 2.1.0 | C | ||
| python27-iris-grib | 0.12.0 | C | |||
| python27-jinja2 | 2.1 | 2.10.1 | C | ||
| python27-jmespath | 0.9.3 | 0.9.4 | C | ||
| python27-joblib | 0.13.0 | 0.13.2 | C | ||
| python27-Jug | 1.6.7 | 1.6.8 | C | ||
| python27-kiwisolver | 1.0.1 | 1.1.0 | C | ||
| python27-latexcodec | 1.0.5 | 1.0.7 | C | ||
| python27-locket | 0.2.0 | 0.2.0 | C | ||
| python27-MarkupSafe | 1.1.0 | 1.1.1 | C | ||
| python27-matplotlib | 3.0.2 | 2.2.2 | C | ||
| python27-mo_pack | 0.2.0 | 0.2.0 | C | ||
| python27-mock | 2.0.0 | 3.0.5 | C | ||
| python27-mpi4py-mpich | 3.0.1 | C | |||
| python27-mpmath | 1.1.0 | 1.1.0 | C | ||
| python27-nappy | 1.1.4 | 1.2.1 | C | ||
| python27-nc-time-axis | 1.1.0 | 1.1.0 | C | ||
| python27-netCDF4 | 1.4.2 | 1.3.1 | C | ||
| python27-nose | 1.3.7 | 1.3.7 | C | ||
| python27-numpy | 1.15.4 | 1.15.4 | C | ||
| python27-packaging | 18.0 | 19.0 | C | ||
| python27-pandas | 0.23.4 | 0.24.2 | C | ||
| python27-partd | 0.3.9 | 1.0.0 | C | ||
| python27-patsy | 0.5.1 | 0.5.1 | C | ||
| python27-pep8 | 1.7.1 | 1.7.1 | C | ||
| python27-Pillow | 5.3.0 | 5.2.0 | C | ||
| python27-psutil | 5.4.8 | 5.6.3 | C | ||
| python27-psycopg2 | 2.7.6.1 | 2.7.7 | C | ||
| python27-pybtex | 0.22.0 | 0.22.2 | C | ||
| python27-pycairo | 1.18.0 | 1.16.3 | C | ||
| python27-pycodestyle | 2.4.0 | 2.5.0 | C | ||
| python27-Pydap | 3.2.2 | 3.2.2 | C | ||
| python27-pygeode | 1.2.2 | C | |||
| python27-Pygments | 2.3.1 | 2.4.2 | C | ||
| python27-pygobject2 | 3.28.3 | C | pygobject in conda | ||
| python27-pygrib | 2.0.3 | 2.0.2 | C | ||
| python27-pygtk2 | C | ||||
| python27-pygtk2-libglade | N | ||||
| python27-pyhdf | 0.9.10 | 0.10.1 | C | ||
| python27-pyke | 1.1.1 | 1.1.1 | C | ||
| python27-pyparsing | 2.3.0 | 2.4.0 | C | ||
| python27-pyproj | 1.9.5.1 | 1.9.6 | C | ||
| python27-pyshp | 2.0.0 | 2.1.0 | C | ||
| python27-pyside | 5.6.0a1 | C | |||
| python27-pyspharm | 1.0.9 | 1.0.9 | C | ||
| python27-pystan | 2.17.1.0 | 2.17.1.0 | C | ||
| python27-pytz | 2018.7 | 2019.1 | C | ||
| python27-pyugrid | 0.3.1 | 0.3.1 | C | ||
| python27-PyYAML | 3.13 | 5.1.1 | C | ||
| python27-pyzmq | 17.1.2 | 18.0.2 | C | ||
| python27-requests | 2.21.0 | 2.22.0 | C | ||
| python27-rpy2 | 2.9.4 | 2.8.5 | C | ||
| python27-s3transfer | 0.1.13 | 0.2.1 | C | ||
| python27-ScientificPython | N | only available in Python2.7 | |||
| python27-scikit-image | 0.14.2 | C | |||
| python27-scikit-learn | 0.20.1 | 0.20.3 | C | ||
| python27-scipy | 1.1.0 | 1.1.0 | C | ||
| python27-scitools-iris | 2.2.0 | 1.13.0 | C | ||
| python27-seaborn | 0.9.0 | C | |||
| python27-setuptools | 40.6.3 | 41.0.1 | C | ||
| python27-Shapely | 1.6.4 | 1.6.4 | C | ||
| python27-singledispatch | 3.4.0.3 | C | |||
| python27-six | 1.12.0 | 1.12.0 | C | ||
| python27-snowballstemmer | 1.2.1 | 1.9.0 | C | ||
| python27-Sphinx | 1.8.2 | 1.8.5 | C | ||
| python27-sphinxcontrib-websupport | 1.1.0 | 1.1.2 | C | ||
| python27-statsmodels | 0.9.0 | 0.10.0 | C | ||
| python27-sympy | 1.3 | 1.4 | C | ||
| python27-Theano | 1.0.3 | 1.0.4 | C | ||
| python27-toolz | 0.9.0 | 0.10.0 | C | ||
| python27-tornado | 5.1.1 | 5.1.1 | C | ||
| python27-tqdm | 4.28.1 | 4.32.2 | C | ||
| python27-typing | 3.7.4 | C | |||
| python27-urllib3 | 1.24.1 | 1.25.3 | C | ||
| python27-virtualenv | 16.0.0 | 16.0.0 | C | ||
| python27-WebOb | 1.8.4 | 1.8.5 | C | ||
| python27-windspharm | 1.7.0 | 1.7.0 | C | ||
| python27-wxPython | 4.0.3 | 4.0.3 | C | ||
| python27-xarray | 0.11.0 | 0.11.3 | C | ||
| qt-devel | 4.8.7 | R | |||
| R | 3.5.1 | 3.4.1 | C | ||
| R-devel | 3.5.1 | 3.4.1 | C | ||
| R-ncdf4 | 1.16 | 1.16 | C | ||
| redhat-lsb | 4.1 | R | |||
| rjags | 4_6 | C | |||
| sqlite-devel | 3.7.17 | R | |||
| subversion | 1.8.17 | R | |||
| subversion-devel | 1.8.17 | R | |||
| subversion-tools | 1.8.17 | R | |||
| tcl-devel | 8.5 | R | |||
| tcl-devel | 8.5.13 | R | |||
| tcsh | 6.18.01 | R | |||
| thea | N | discontinued | |||
| tk | 8.6.9 | 8.6.9 | 8.5.13 | CR | |
| tk-devel | 8.6.9 | 8.6.9 | 8.5.13 | CR | |
| tkdiff | 4.3.5 | R | |||
| tmux | 2.7 | 2.7 | C | ||
| tree | 1.6.0 | R | |||
| udunits-devel | 2.2.27.6 | 2.2.27.6 | 2.2.20 | R | called |
| udunits2-devel | |||||
| umutil | 20130102 | R | |||
| umutil-lib | 20130102 | R | |||
| uuid | 1.6.2 | R | |||
| uuid-devel | 1.6.2 | R | |||
| valgrind | 3.15.0 | C | |||
| vim-enhanced | 7.4.629 | R | |||
| wxGTK-devel | 2.8.12 | R | |||
| xconv | 1.94 | R | |||
| xemacs | 21.5.34 | R | |||
| xorg-x11-util-macros | 1.19.0 | R | |||
| xpdf | 3.04 | R |