Creating a virtual environment in the jasmin notebooks service

Creating a virtual environment in the JASMIN Notebooks service.

This article describes:

how to create a virtual environment
how to make the notebook service recognise your new environment as a kernel
how use your new kernel
other tips & useful knowledge

Creating a virtual environment is useful to allow a discrete set of extra packages to be installed to meet specific requirements. This allows a user to run multiple environments with different dependencies without conflicts.

There are a number of ways to create a virtual environment to use with the notebook service. This document outlines the most common and reccomended methods, and then some other ways which you might find useful.

Please note that environments created for the notebook service will not work on the jasmin scientific analysis servers or the LOTUS batch processing.

Step 1: Creating a virtual environment

This step creates a python virtual environment, and allows you to install packages into it. These commands are intended for use at the jupyter notebooks' shell not on the JASMIN sci machines

To get started, head to https://notebooks.jasmin.ac.uk and click the terminal button.

Then, type these commands at the bash shell which appears.

First, make a directory in which to store your virtual environments. You can put this wherever you like, as long as you reference the same place later. You could store several virtual environments within this directory, for different purposes. Then, change into that directory.

mkdir ~/nb_envs
cd ~/nb_envs

Next, create a new empty virtual environment. We reccomended including the --system-site-packages argument which will allow you to add packages on top of jaspy, rather than starting completely from scratch.

python -m venv name-of-environment --system-site-packages

Then, activate the specific virtual environment created above, which will allow you to install packages.

source name-of-environment/bin/activate

If you want to be able to use your virtual environment as a jupyter notebook kernel (reccomended), you should install

ipykernel

using pip. bash pip install ipykernel

You can then install whatever packages you need in the environment.

pip install pyjokes

If you change your mind and need to add more packages in the future, it is simple to activate the virtual environment in the same way as above and use pip to install more packages.

Step 2: Making the notebook service recognise your new kernel.

These steps are also run from the notebooks' service shell, as above.

If you aren't still there from the last step, cd to the location of your venv.

cd ~/nb_envs

If it isn't already active, activate the virtual environment.

source name-of-environment/bin/activate

Running the following command will make the notebook's service notice your new virtual environment, and include it in the list of kernels which you can run code with. You only have to do this once.

python -m ipykernel install --user --name=name-of-environment

Step3: Using Your New Kernel

You can then choose this kernel from the jupyterhub homepage, or from the top right of any open notebook. No changes to the python code within are required.

Other tips & useful knowledge

Activating an environment without it being a kernel.

If you follow Step 1 above to create a virtual environment, it is possible to use the packages from this environment in a python file without making it a kernel. While this can be useful, it has the very distinct disadvantage of hardcoding the path to your virtual environment in your python code. For this reason we discourage using this method with a medium level of severity. To do this, simply add the following code to your python file before any imports. Adjust the venv_path variable to be correct for the venv you created.

import sys
import pathlib
import platform

venv_path = "~/nb_envs/name-of-environment"

py_version = platform.python_version_tuple()
sys.path.append(
    str(
        pathlib.Path(
            f"{venv_path}/lib/python{py_version[0]}.{py_version[1]}/site-packages/"
        ).expanduser()
    )
)

Explanation: this adds the site-packages folder from your venv directly to the path python uses to search for packages (yourPYTHONPATH`). This lets python find them to import.

Can I install packages from inside my python code?

We very strongly recomend NOT trying to install python packages from inside notebook code. pip isn't designed for it, and it is almost always easier to activate the venv as above and install things that way.

If you wish to record the set of packages inside your venv so you can install them en-masse later, pip has the facility to do this. To export a list of packages that exist inside a venv, from the notebooks bash shell with the virtual environment in question activated:

pip freeze > requirements.txt

To install a list of packages which have been exported:

pip install -r requirements.txt

Exporting packages in this way is also useful for sharing your environment with others, reinstalling when it breaks etc. It's a good idea to keep the requirements file alongside the code in version control. If your code becomes more complex it is probably more sensible to make it a python package, and install it as one, but doing that is outside the scope of this document.

If you really must, you can call pip from inside your notebook like this: (after first updating the packages variable to be the ones you want to install.)

import sys
import subprocess as sp

packages = ['pyjokes']

sp.check_call([sys.executable, '-m', 'pip', 'install'] + packages)

Can I use conda instead of a virtual environment?

Yes, no problem.

To create a conda environment, sinply run the following at the JASMIN Notebooks shell:

conda create --name name-insert-here ipykernel

Install any packages you which to use in the environment:

conda install --name name-insert-here pyjokes

Make the notebook service recognise your environment as a kernel:

conda run --name name-insert-here python -m ipykernel install --user --name name-insert-here

Can get rid of old kernels from the notebook service?

Yes.

To list the names of kernels you have installed, run the following at the JASMIN notebook's shell:

jupyter kernelspec list

To remove one of them, run:

jupyter kernelspec uninstall insert-name-here