Data Transfer Tools: rsync, scp, sftp
This article tells you about some of the basic transfer tools available for use with JASMIN that work over an SSH connection:
- rsync (over SSH)
- scp
- sftp
rsync over SSH
$ rsync mydata user@remote-host:/data/
will use SSH to connect as user
to remote-host
. Once connected, it will invoke another copy of rsync on the remote host, and then the two programmes will talk to each other over the connection, working together to determine which parts of mydata
are already on the remote host and don't need to be transferred over the connection.
The generic syntax is:
$ rsync [OPTION] ... SRC [SRC] ... [USER@]HOST:DEST $ rsync [OPTION] ... [USER@]HOST:SRC [DEST]
...where SRC is the file or directory (or a list of multiple files and directories) to copy from, and DEST represents the file or directory to copy to. (Square brackets indicate optional parameters.)
For more information visit the official rsync website.
rsync example on JASMIN
Here is a simple example using rsync over SSH to copy a file to a Group Workspace on JASMIN:
$ exec ssh-agent $SHELL $ ssh-add ~/.ssh/id_rsa_jasmin # or other key filename as needed $ rsync myfile <username>@xfer1.jasmin.ac.uk:/group_workspaces/jasmin/myproject/data/
NOTE: The first two lines are the standard method for setting up the SSH agent in order to allow connections without prompting for a passphrase each time.
scp
scp
is another basic command-line tool for secure copying between two machines. It is installed as part of most SSH implementations and comes as standard on the JASMIN transfer servers. scp
is the secure analogue of the rcp
command. Typically, the syntax of scp
is like the syntax of cp
(copy):
To copy a file to a host:
$ scp myfile user@host:directory/target
To copy a file (or directory) from a host to the local system:
$ scp user@host:directory/source target $ scp -r user@host:directory/source_folder target_folder
Note that if the remote host uses a port other than the default of 22, it can be specified in the command. For example, copying a file from host over a non-standard SSH port (2222):
$ scp -P 2222 user@host:directory/source target
For more information on scp please visit the following website.
sftp
sftp is a similar command-line tool to scp, but the underlying SFTP protocol allows for a range of operations on remote files which make it more like a remote file system protocol. sftp includes extra capabilities such as resuming interrupted transfers, directory listings, and remote file removal.
For basic transfer of a file on JASMIN to the local machine:
$ sftp user@xfer1.jasmin.ac.uk:/group_workspaces/jasmin/myproject/data/notes.txt ./
For more information see the Wikipedia page on SFTP.
There are various 3rd-party tools and clients, for example, WinSCP, FileZilla, MobaXterm and others, which can do transfers using the SCP and/or SFTP protocols. Please refer to the documentation for that particular software, but the basic principles are the same, i.e that you can set up a connection to a remote host by specifying the same parameters that you would with the command-line tool, but often through a graphical user interface instead.
Note on performance
While convenient and familiar to many users, the tools described above do not make efficient use of available bandwidth for transferring large quantities of data via high-speed networks over long distances. Please consult Data Transfer Tools to learn more about which might be the most appropriate tool to use in different contexts.