Transfers to/from ARCHER/RDF

This article explains how to transfer data between JASMIN and ARCHER/RDF. It covers:

  • The choice of available tools / routes
  • How to request that the RDF transfer server "trusts" a credential issued by CEDA's short-term credential server

Choice of available Tools/Routes

See  JASMIN external connections and Data Transfer Tools  for general details.

It is important to choose the most appropriate route, method and tools to ensure you get the most efficient and reliable transfer experience. This can vary depending on system and network conditions, so it is worth experimenting with the different options available.

If you want to try  all the options available, you will need:

  • high-performance data transfer access on JASMIN
  • a login account at ARCHER/RDF, with access to the Data Transfer Nodes (dtn)
  • to have registered the subject of your CEDA-issued short-term credential with ARCHER support.

Check the examples in the linked documentation articles and ensure that you use them between the hosts used in the examples. Not all services connect over all routes to/from all hosts!

Table 1, below, shows recommended combinations of hosts & tools for transfers between RDF and JASMIN.

scp/rsync/sftp
Simple transfer using easy method but poor use of bandwidth
source dest notes
jasmin-xfer1 dtn02 over 1G light path
jasmin-xfer2 dtn0[12] over 10G JANET
bbcp
More sophisticated transfer tool, better use of bandwidth
jasmin-xfer1 dtn02 over 1G light path
jasmin-xfer2 dtn0[12] over 10G JANET
GridFTP over SSH   GridFTP performance with convenience of SSH, but only simple features of GridFTP
jasmin-xfer1 dtn02 over 1G light path
jasmin-xfer2 dtn0[12] over 10G JANET
GridFTP using certificate auth
Fully-featured GridFTP. Best performance.
jasmin-xfer1 dtn02 over 1G lightpath, client at JASMIN
jasmin-xfer2 dtn0[12] over JANET 10G, client at JASMIN
data-xfer1 dtn0[12] over JANET 10G, client at RDF
GridFTP using Globus Online
Managed GridFTP transfer using  web GUI, CLI or API. Best performance, reliable transfers.
JASMIN gridftp server Globus endpoint (data-xfer1)
ARCHER RDF Globus endpoint (dtn03)
over 10G JANET

Table 1: comparison of methods and routes for transferring data between RDF and JASMIN.

Recommended method: Gridftp using Globus Online

The recommended method is to use Globus Online to manage the transfer of data.

The following YouTube video shows this in action, although a few pre-requisites are needed, which are shown below:

This shows the following steps:

  1. Navigating to the Globus web interface
  2. Locating the "ARCHER RDF" Globus endpoint, and activating it
  3. Locating the "JASMIN gridftp server" and activating it
  4. Initiating a transfer of a file between the 2 endpoints
  5. Refreshing the file listing on the destination

For (2), you need to have an account at ARCHER/RDF and need to use those login credentials to authenticate with that endpoint. Queries relating to this account this should be directed to support@archer.ac.uk.

For (3), you need to have a JASMIN account, with hpxfer access, but need to use the credentials of your linked CEDA account in order to authenticate with the endpoint. Queries relating to this account should be directed to support@ceda.ac.uk.

How to request that the RDF transfer node(s) trust a credential issued by CEDA's short-term credential server

(Only needed for method "Gridftp using certificate auth". Should not be needed for Globus Online transfer)

  • Go to https://www.archer.ac.uk/safe/
  • Go to "login accounts" and click your username
  • Click "Add certificate"
  • Enter the Distinguishing Name (DN) of your CEDA-issued short-term credential. You can obtain this by using the following commands:
$ myproxy-logon -s slcs1.ceda.ac.uk -l USERNAME -o credfile

The first time you do this, or in the case where host certificates have been updated, add the -b option to ensure that the relevant certificate files are copied to your local certificate store.

$ myproxy-logon -s slcs1.ceda.ac.uk -l USERNAME -o credfile -b

In this case, the output, i.e. the short-term credential, will be stored in credfile. Inspect this file and extract the subject:

$ openssl x509 -noout -in credfile -subject
subject= /DC=uk/DC=ac/DC=ceda/O=STFC RAL/CN=https://ceda.ac.uk/openid/Firstname.Lastname
  • Copy and paste the text beginning /DC= into the appropriate place in the form presented by the SAFE interface and submit the request.
  • Once you hear back from support@archer.ac.uk that your request has been completed, your credential should be trusted and you should be authorized to use the transfer server in this way. There is a chance that by this time, your short-term credential will have expired (!). If so, repeat the myproxy-logon step again, but when re-issued it will have the same subject and should be ready to use. Note also that there is sometimes a delay between the time when you receive the notification message from Archer support and the next time that their cron job runs to roll out the change to their servers. If in doubt, wait a few hours and try again.

Still need help? Contact Us Contact Us