Data Transfer Tools: rsync, scp, sftp

This article tells you about some of the basic transfer tools available for use with JASMIN that work over an SSH connection:

rsync (over SSH)
scp
sftp

rsync over SSH

rsync is a file synchronisation and file transfer program for Unix-like systems that can minimise network data transfer by using a form of delta encoding such that only the differences between and source and destination data are actually transmitted. rsync can compress the data transferred further using zlib compression and SSH or stunnel can be used to encrypt the transfer.

rsync is typically used to synchronise files and directories between two different systems, one local and one remote. For example, the command:

$ rsync mydata user@remote-host:/data/

will use SSH to connect as user to remote-host. Once connected, it will invoke another copy of rsync on the remote host, and then the two programmes will talk to each other over the connection, working together to determine which parts of mydata are already on the remote host and don't need to be transferred over the connection.

The generic syntax is:

$ rsync [OPTION] ... SRC [SRC] ... [USER@]HOST:DEST 
$ rsync [OPTION] ... [USER@]HOST:SRC [DEST]

...where SRC is the file or directory (or a list of multiple files and directories) to copy from, and DEST represents the file or directory to copy to. (Square brackets indicate optional parameters.)

For more information visit the official rsync website.

rsync example on JASMIN

Here is a simple example using rsync over SSH to copy a file to a Group Workspace on JASMIN:

$ exec ssh-agent $SHELL
$ ssh-add ~/.ssh/id_rsa_jasmin  # or other key filename as needed
$ rsync myfile <username>@xfer1.jasmin.ac.uk:/group_workspaces/jasmin/myproject/data/

NOTE: The first two lines are the standard method for setting up the SSH agent in order to allow connections without prompting for a passphrase each time.

scp

scp is another basic command-line tool for secure copying between two machines. It is installed as part of most SSH implementations and comes as standard on the JASMIN transfer servers. scp is the secure analogue of the rcp command. Typically, the syntax of scp is like the syntax of cp (copy):

To copy a file to a host:

$ scp myfile user@host:directory/target

To copy a file (or directory) from a host to the local system:

$ scp user@host:directory/source target
$ scp -r user@host:directory/source_folder target_folder

Note that if the remote host uses a port other than the default of 22, it can be specified in the command. For example, copying a file from host over a non-standard SSH port (2222):

$ scp -P 2222 user@host:directory/source target

For more information on scp please visit the following website.

sftp

sftp is a similar command-line tool to scp, but the underlying SFTP protocol allows for a range of operations on remote files which make it more like a remote file system protocol. sftp includes extra capabilities such as resuming interrupted transfers, directory listings, and remote file removal.

For basic transfer of a file on JASMIN to the local machine:

$ sftp user@xfer1.jasmin.ac.uk:/group_workspaces/jasmin/myproject/data/notes.txt ./

For more information see the Wikipedia page on SFTP.

There are various 3rd-party tools and clients, for example, WinSCP, FileZilla, MobaXterm and others, which can do transfers using the SCP and/or SFTP protocols. Please refer to the documentation for that particular software, but the basic principles are the same, i.e that you can set up a connection to a remote host by specifying the same parameters that you would with the command-line tool, but often through a graphical user interface instead.

Note on performance

While convenient and familiar to many users, the tools described above do not make efficient use of available bandwidth for transferring large quantities of data via high-speed networks over long distances. Please consult Data Transfer Tools to learn more about which might be the most appropriate tool to use in different contexts.