Step-by-Step guide to setting up the NLDS client on JASMIN

Changes from previous method  

• The access and secret keys required by the object store are now automatically generated by the nlds init command. The previous workflow of generating the object store access and secret keys is no longer required.
• In addition, the nlds init command will now fill in all of the required values in the ~/.nlds-config file, including the user and group. This makes the setting up of the NLDS client much more user-friendly.
• PyPi is now used for hosting the NLDS client, which makes installation much more straightforward for the user. pip install nlds-client can now be used, rather than the long GitHub URL as previously.
• JASMIN users are automatically granted access to the nlds-cache-01-o object store tenancy on JASMIN. They no longer have to apply for access to this tenancy via the JASMIN accounts portal.

Prerequisites  

This guide assumes that the user already has:

1. A JASMIN user account and the jasmin-login access role. See Getting Started.
2. Access to at least one Group Workspace. See Apply for access to a group workspace.

With those in place, the remainder of this guide will take you through the following additional steps to access NLDS as a JASMIN user:

1. Getting access to the NLDS object storage.
2. Installing the NLDS client.
3. Running the nlds init command.

Access to the object storage  

Data pushed to the NLDS service is initially written to a cache on object storage. Similarly, when data is brought back from tape, it is initially written to the object store cache before being made available to you.

Normally, to gain access to object storage ( HPOS or s3 storage) on JASMIN, you would need to apply for an additional access role via the JASMIN Accounts Portal. However, this has now been automated for the object storage used for the NLDS cache, so this is now tied to your jasmin-login access role.

For further information about object storage on JASMIN, see Using the JASMIN Object Store.

Installing the NLDS client in your JASMIN home directory  

Installing the NLDS client in your home directory enables use of the nlds client in the following recommended places on JASMIN:

• sci servers
• transfer servers (xfer-vm-0x)

Please read the documentation about each type of server.

Creating the required virtual environment in your home directory will ensure that it is available from both types of server.

1. Log into your chosen server:

   ssh <user>@sci-vm-01.jasmin.ac.uk

   Sci servers usually need to be accessed via a login server if you are outside the STFC network. Transfer servers can be accessed directly .See also login servers

2. Install the NLDS client software:

   Full documentation here  .

   The essential steps are reproduced below:

   cd $HOME           # ensure that you are in your home directory
   module load jaspy  # ensure that you're using the current Python environment
   python3 -m venv ~/nlds-client
   source ~/nlds-client/bin/activate
   pip install --upgrade pip
   pip install nlds-client

Configuring the NLDS client  

1. First make sure your are running the nlds-client virtual-environment that you created above

   (Tip: try echo $VIRTUAL_ENV: is the returned path correct?).

   If you are not running it then issue the command:

   source ~/nlds-client/bin/activate

2. Run NLDS with the init command to create your NLDS config file.

   This file will be in your home directory with the name .nlds-config. The full path is ~/.nlds-config, also $HOME/.nlds-config.

   Use the -g|--group option to specify the default Group Workspace (GWS) when you are working with NLDS. This should be the short name e.g. myproject and NOT the corresponding unix group name with the gws_ prefix, e.g. gws_myproject.

   nlds init -g <default gws>
   Initialising the Near-line Data Store...
   • This application uses OAuth2 to authenticate with the server on your behalf.
   • To do this it needs your password.  Your password is not stored.
   • It is used to obtain an access token, which is stored in the file: ~/.nlds-token,    (out)and used for subsequent interactions with the server.
   • It is also used to obtain object storage keys.  These are stored in the configuration file: ~/.nlds-config and used for interaction with the object storage    (out)cache.
   
   Password:

   Note the output shown above, with the password prompt at the end.

   Enter your JASMIN accounts portal password. The user name is set automatically.

   You should now see the following message:

   Successfully initialised, the config file at ~/.nlds-config has been
   updated with all of the necessary information to start using the NLDS.

   You can now examine the config file with the following command (use your preferred editor, nano is used here):

   nano ~/.nlds-config

   or

   less ~/.nlds-config

   if you simply want to read it: there should be no need to modify it.

   The file should look like the example below. Fields between < > characters should have values in them. If these fields contain the characters {{ }} then please delete the ~/.nlds-config file and try nlds init -g <gws> again. If this fails again then please contact the JASMIN helpdesk.

   {
       "server": {
           "url": "https://nlds.jasmin.ac.uk",
           "api": "api/1.0.0"
       },
       "user": {
           "default_user": "< your JASMIN username >",
           "default_group": "< your default GWS >"
       },
       "authentication": {
           "oauth_client_id": "< redacted >",
           "oauth_client_secret": "< redacted >",
           "oauth_token_url": "< redacted >",
           "oauth_scopes": "< redacted >",
           "oauth_token_file_location": "~/.nlds-token"
       },
       "object_storage": {
           "tenancy": "nlds-cache-01.s3.jc.rl.ac.uk",
           "access_key": "< redacted >",
           "secret_key": "< redacted >"
       },
       "options": {
           "verify_certificates": true
       }
   }

Running the NLDS client for the first time  

You should now have a fully configured and functioning NLDS client. You can now run the NLDS client for the first time. Make sure the Python virtual environment you created earlier is activated first, then run nlds with the command stat.

You should see something similar to this:

source ~/nlds-client/bin/activate
nlds stat
Usage: nlds stat [OPTIONS]
Try 'nlds stat --help' for help.

Error: Failed to get status of transaction(s) with user:<user>, group:<gws>
Reason: no transaction records found for user:< user > and group:< gws >

Because you issued the stat command with nlds, the status of all your NLDS requests will be shown. This is empty, as this is the first time you have run NLDS, and the error message reflects this by stating no transaction records found.

Next steps  

• Basic usage
• Full NLDS tutorial 

Now that you have the NLDS client set up on JASMIN, you can go through the tutorial in the NLDS client documentation. This includes how to use the commands listed with the nlds --help command:

nlds --help

Usage: nlds [OPTIONS] COMMAND [ARGS]...

Options:
-v, --version  Output NLDS client version and exit.
--help         Show this message and exit.

Commands:
find     Find and list files.
get      Get a single file.
getlist  Get a number of files specified in a list.
init     Set up the NLDS client on first use.
list     List holdings.
meta     Alter metadata for a holding.
put      Put a single file.
putlist  Put a number of files specified in a list.
stat     List transactions.