Issue affecting data transfers using Globus Online
This article provides current advice to JASMIN users of Globus Online and other high-performance transfer methods which use parallel streams.
An issue has been affecting some data transfer methods for several months. The JASMIN team is still investigating the cause and how to address it, but our current advice is that transfers to storage other than
/gws/smf/ via Globus Online, or via other methods using parallel streams, is not recommended. Other methods (bbcp, gridftp) may be OK if single-stream transfers can be specified. Single-stream methods are unaffected.
- transfers to
/gws/smfstorage seem unaffected
- transfers to
/gws/nopwstorage are affected, with performance to /gws/nopw storage areas particularly impaired.
Read operations are not affected, so you can safely use Globus Online and other parallel-stream methods for transferring data from all storage on JASMIN.
What to do for now?
globus-url-copy, you must omit the
-p N -fastoptions which are used to enable parallel-stream transfers (do not simply set N to 1). The
-cc N(concurrency) option is still safe to use, as this initiates N single-stream transfers, one for each file (as opposed to multiple streams to transfer a single file, which is problematic). Please see transfers from ARCHER2 for an example of how to do this, for single and multiple files at once.
bbcp, the default number of streams is 4, so you must actively set this to 1 with the
sftpwhich only do single stream transfers (these are convenient but do not achieve the best performance over a given route)