Near-line storage for group workspace data

Introduction  

The Near-Line Data Store is JASMIN’s new service to bridge the gap between disk and tape storage.

It makes use of a large cache on Object Storage to help stage data efficiently to and from a large-capacity, high-throughput tape library at the backend. It provides significantly more functionality than previous generations of near-line storage services, so please take the time to find out more.

NLDS aims to provide:

• a more dynamic way of using tape in users’ workflows
• more energy-efficient storage, important for both cost and environmental impact of your work
• all JASMIN users with a means to store data near-line (previously it was aimed more at GWS managers only)

Full documentation is available here: NLDS documentation     

Webinar  

Our webinar in March 2025 gave an overview of NLDS: please watch this to find out more. The NLDS part of the presentation lasts around 33 minutes.

Basic usage  

You’ll need to install the client software  to use the commands, but some examples are:

PUT a single file into a holding:

nlds put <filepath>

(you can PUT a list of files, too).

Check on the status of your holdings:

nlds stat
State of transactions for user:frjohn, group:farmers
    user        id    action          job label       label           done  state                  last update
    frjohn      1     put             SheepHerding    SheepPen        100%  COMPLETE               2023-04-18 15:21:41
    frjohn      2     put             test_putlist    Zoo             100%  COMPLETE

Retrieve a file from a holding:

First, find the holding that the file is stored in:

nlds find
Listing files for holdings for user:frjohn, group:farmers
    user        h-id  h-label         size    date        storage path
    frjohn      1     SheepPen        49.0B   2023-04-18     O    /Users/frjohn/sheep.txt
    frjohn      2     Zoo             96.0B   2023-04-18     O    /Users/frjohn/albatross.txt
    frjohn      2     Zoo             50.0B   2023-04-18     L    /Users/frjohn/rabbit.txt

Fetch a single file:

nlds get /Users/frjohn/sheep.txt -r ./
GETLIST transaction accepted for processing.
    user            : frjohn
    group           : farmers
    action          : getlist
    job label       : 14bc9846
    transaction id  : 14bc9846-9d45-440a-af6c-dfcb5cb9dcae

Further information  

NLDS is fully documented here  , including:

• Tutorial
• Step-by-step guide to setup
• Hints and Tips