Joint-storage Data Migration App (JDMA)

See the JDMA user documentation at: cedadev.github.io/jdma_client  for more information about using JDMA.

The joint-storage data migration app (JDMA) is a multi-tiered storage system which provides a single API to users to allow the movement of data to a number of different storage systems, query the data they have stored on those storage systems and retrieve the data.

These interactions are carried out using a common user interface, which is a command line tool to be used interactively, a python library or a HTTP API, both to be used programmatically. The command line tool essentially provides a wrapper for calls to the python library, which in turn makes calls to the HTTP API.

JDMA was designed with the following usability criteria in mind:

• The user experience for moving data, regardless of the underlying storage systems, should be identical.
• The user should not be responsible for maintaining the connection to the storage system in the case of asynchronous transfers.
• User and group ownership and permissions should be preserved and restored on downloading the data
• The user should receive notifications when the transfers are complete.
• Users should be able to transfer data from one storage system to another
• JDMA is only a request and query layer. Any cataloguing of data should be carried out by the backend system. So that, if JDMA fails, then the data is still available independently of JDMA, from the storage backend.

See the JDMA user documentation at: cedadev.github.io/jdma_client/  for more information about using JDMA.

JDMA was development under a Horizon 2020 grant from the EU Commission. A report submitted to the EU Commission can be found in the repository at: [github.com/cedadev/django-jdma_control/blob/master/doc/ESiWACE- Milestone-8_final.pdf]( https://github.com/cedadev/django-  jdma_control/blob/master/doc/ESiWACE-Milestone-8_final.pdf)

The JDMA client github is at: github.com/cedadev/jdma_client 

Quick guide to installing the JDMA client on JASMIN  

If you are working on JASMIN and you wish to use the JDMA client, then you can install it as follows on a sci server:

module load jaspy
python -m venv ~/venvs/jdma-venv
source ~/venvs/jdma-venv/bin/activate
pip install git+https://github.com/cedadev/jdma_client

You should then have the jdma command-line tool available in your terminal session.

source ~/jdma_venv/bin/activate
pip install --upgrade git+https://github.com/cedadev/jdma_client
pip list | grep jdma-client