Near-line storage for group workspace data

Introduction  

The Near-Line Data Store is JASMIN’s new service to bridge the gap between disk and tape storage.

It makes use of a large cache on Object Storage to help stage data efficiently to and from a large-capacity, high-throughput tape library at the backend. It provides significantly more functionality than previous generations of near-line storage services, so please take the time to find out more.

NLDS aims to provide:

• a more dynamic way of using tape in users’ workflows
• more energy-efficient storage, important for both cost and environmental impact of your work
• all JASMIN users with a means to store data near-line (previously it was aimed more at GWS managers only)

Documentation is available here:

• step by step guide to setup on JASMIN
• basic usage (this document)
• Full NLDS tutorial  as part of NLDS client software documentation.

It is recommended to start with the step-by-step guide

Basic usage  

You’ll need to follow the step by step guide to use these commands on JASMIN, but some examples are:

PUT a single file into a holding:

nlds put <filepath>

(you can PUT a list of files, too).

Check on the status of your holdings:

nlds stat
State of transactions for user:frjohn, group:farmers
    user        id    action          job label       label           done  state                  last update
    frjohn      1     put             SheepHerding    SheepPen        100%  COMPLETE               2023-04-18 15:21:41
    frjohn      2     put             test_putlist    Zoo             100%  COMPLETE

Retrieve a file from a holding:

First, find the holding that the file is stored in:

nlds find
Listing files for holdings for user:frjohn, group:farmers
    user        h-id  h-label         size    date        storage path
    frjohn      1     SheepPen        49.0B   2023-04-18     O    /Users/frjohn/sheep.txt
    frjohn      2     Zoo             96.0B   2023-04-18     O    /Users/frjohn/albatross.txt
    frjohn      2     Zoo             50.0B   2023-04-18     L    /Users/frjohn/rabbit.txt

Fetch a single file:

nlds get /Users/frjohn/sheep.txt -r ./
GETLIST transaction accepted for processing.
    user            : frjohn
    group           : farmers
    action          : getlist
    job label       : 14bc9846
    transaction id  : 14bc9846-9d45-440a-af6c-dfcb5cb9dcae

Further information  

NLDS is fully documented here  , including:

• Tutorial
• Step-by-step guide to setup
• Hints and Tips

Webinar  

Our webinar in March 2025 gave an overview of NLDS: please watch this to find out more. The NLDS part of the presentation lasts around 33 minutes.