Docs
Transfers from ARCHER2
Transferring data from ARCHER2 to JASMIN, efficiently
Choice of available Tools/Routes
See Data Transfer Tools for general information.
Users transferring data between ARCHER2 and JASMIN are often transferring relatively large sets of data, so it is important to choose the most appropriate route, method and tools to ensure you get the most efficient and reliable transfer experience. This can vary depending on system and network conditions.
The recommended option (as of mid-2024) is now Globus.
- See here for a quick-start guide to using Globus with JASMIN.
- Further aspects specific to transfers between ARCHER2 and JASMIN are covered below.
Common requirements to all of the methods are:
- an account with the jasmin-login access role on JASMIN.
- a login account at ARCHER2
Please note:
- Enquiries about access to or use of ARCHER2 should be directed to ARCHER2 support ( support@archer2.ac.uk)
- Enquiries about access to or use of JASMIN should be directed to JASMIN support (use beacon, below-right or support@jasmin.ac.uk)
Available transfer methods
- Globus (recommended)
- Basic SSH transfer (slow but convenient)
1st choice method: Globus
This is now the recommended method, because:
- it always ensures the best performance
- it is a managed transfer service, less prone to overloading and system issues
- it is actively maintained
- it is easy to use
Please read our quick-start guide to using Globus with JASMIN first, then come back here.
Because Globus can do transfers between two third-party locations, you don’t necessarily need to invoke the transfers from a machine on JASMIN, or ARCHER2 (even though it’s those two locations which will be involved as source and destination for the transfer). This could be done from your laptop or desktop, but could also be done from within a workflow that’s running somewhere (e.g. ARCHER2 or JASMIN). So, first think about where you want to control the process from.
In that location, follow the steps below:
1. Set up the Globus Command Line interface
- follow the steps described here
2. Identify the collections that you want to transfer between, for your transfer:
In this case, these are likely to be:
- the
ARCHER2 filesystems collection
, with ID
3e90d018-0d05-461a-bbaf-aab605283d21 - the
JASMIN default collection
, with ID
a2f53b7f-1b4e-4dce-9b7c-349ae760fee0
Set an environment variable for each of these, to avoid having to type the ID each time:
export a2c=3e90d018-0d05-461a-bbaf-aab605283d21
export jdc=a2f53b7f-1b4e-4dce-9b7c-349ae760fee03. Check access to these collections
These collections are restricted-access rather than public, so your access to them is via a series of authentication/authorisation/consent steps which the following actions will guide you through:
globus ls $a2c:/~/
(ARCHER2 home directory file listing should appear)globus ls $jdc:/~/
(JASMIN home directory file listing should appear)The steps above establish your ability to interact with each of the specified collections using Globus. Once you’ve completed each one, you should see a directory listing.
In both cases, if you don’t yet have consent and authentication for that particular collection, you will be provided with the suggested command to run in your terminal environment:
globus ls $a2c$:
The collection you are trying to access data on requires you to grant consent for the Globus CLI to access it.
Please run:
globus session consent 'urn:globus:auth:scope:transfer.api.globus.org:all[*https://auth.globus.org/scopes/3e90d018-0d05-461a-bbaf-aab605283d21/data_access]'
to login with the required scopes.
Run the suggested command:
globus session consent 'urn:globus:auth:scope:transfer.api.globus.org:all[*https://auth.globus.org/scopes/3e90d018-0d05-461a-bbaf-aab605283d21/data_access]'
You are running 'globus session consent', which should automatically open a browser window for you to authenticate with specific identities.
If this fails or you experience difficulty, try 'globus session consent --no-local-server'
---and complete the authentication process in the browser window.
The equivalent command for the JASMIN Default Collection is this:
globus session consent 'urn:globus:auth:scope:transfer.api.globus.org:all[*https://auth.globus.org/scopes/a2f53b7f-1b4e-4dce-9b7c-349ae760fee0/data_access]'Once you’ve completed these steps for both source and destination collections, the globus ls commands you tried earlier should now work, and you are now ready to try a transfer.
4. Initiate a simple transfer
globus transfer $a2c:/~/1M.dat $jdc:/~/1M.dat
Message: The transfer has been accepted and a task has been created and queued for execution
Task ID: aa0597a4-80a7-11ef-b36b-a1206a7ee65fThis should complete quite quickly for a small file, but for a larger file you can check the progress using the task ID.
globus task show aa0597a4-80a7-11ef-b36b-a1206a7ee65f
Label: None
Task ID: aa0597a4-80a7-11ef-b36b-a1206a7ee65f
Is Paused: False
Type: TRANSFER
Directories: 0
Files: 1
Status: SUCCEEDED
Request Time: 2024-10-02T10:18:32+00:00
Faults: 0
Total Subtasks: 2
Subtasks Succeeded: 2
Subtasks Pending: 0
Subtasks Retrying: 0
Subtasks Failed: 0
Subtasks Canceled: 0
Subtasks Expired: 0
Subtasks with Skipped Errors: 0
Completion Time: 2024-10-02T10:18:39+00:00
Source Endpoint: Archer2 file systems
Source Endpoint ID: 3e90d018-0d05-461a-bbaf-aab605283d21
Destination Endpoint: JASMIN Default Collection
Destination Endpoint ID: a2f53b7f-1b4e-4dce-9b7c-349ae760fee0
Bytes Transferred: 1000000
Bytes Per Second: 148452If you wanted to use the above in a script, and block/wait for the transfer task to complete before
continuing, you can use globus task wait <taskid>, for example:
globus task wait aa0597a4-80a7-11ef-b36b-a1206a7ee65fwill now return control immediately, since the task has completed.
Globus transfer tasks are asynchronous, submitted to your own mini-queue, where you can have as many queued tasks as you like but only 3 in progress at any one time. This ensures good performance for all users, but your tasks do not linger in long multi-user queues. The best way to reassure yourself of this is to try it out.
For help with any globus command you can do globus <command> --help.
Further examples including sync and automation are given in Globus command line interface, with further examples in the Globus documentation at https:// .
Relevant examples:
- sync with wait using the CLI.
- Repeatable transfer using the PythonSDK (more advanced)
Note that Globus transfers (and other actions) can be managed & monitored by:
- a web interface
- the command-line interface, and
- a Python library
all of which interact with the same underlying service.
NCAS-CMS users should note that work is currently underway to adopt Globus as a drop-in replacement for certificate-based gridftp in Rose suites currently in use for automating processing and transferring to JASMIN.
2nd choice method: Basic SSH transfer
scp/rsync/sftp: Simple transfers using easy method, pushing data to general purpose xfer nodes. Convenient, but limited performance.
| source | dest | notes |
|---|---|---|
login.archer2.ac.uk |
xfer-vm-0[123].jasmin.ac.uk |
to virtual machine at JASMIN end |
login.archer2.ac.uk |
hpxfer[34].jasmin.ac.uk |
to high-performance physical machine at JASMIN end |