JASMIN Help Site logo JASMIN Help Site logo
  • Docs 
  • Guides 
  • Training 
  • Discussions   

  •   Search this site  

Can't find what you're looking for?

Try our Google custom search, across all JASMIN sites

Docs
  • getting started
    • get started with jasmin
    • generate ssh key pair
    • get jasmin portal account
    • get login account
    • beginners training workshop
    • how to contact us about jasmin issues
    • jasmin status
    • jasmin training accounts
    • tips for new users
    • how to login
    • multiple account types
    • present ssh key
    • reconfirm email address
    • reset jasmin account password
    • ssh auth
    • storage
    • understanding new jasmin storage
    • update a jasmin account
  • interactive computing
    • interactive computing overview
    • check network details
    • login servers
    • login problems
    • graphical linux desktop access using nx
    • sci servers
    • tenancy sci analysis vms
    • transfer servers
    • jasmin notebooks service
    • jasmin notebooks service with gpus
    • creating a virtual environment in the notebooks service
    • project specific servers
    • dask gateway
    • access from vscode
  • batch computing
    • lotus overview
    • slurm scheduler overview
    • slurm queues
    • lotus cluster specification
    • how to monitor slurm jobs
    • how to submit a job
    • how to submit an mpi parallel job
    • example job 2 calc md5s
    • orchid gpu cluster
    • slurm status
    • slurm quick reference
  • software on jasmin
    • software overview
    • quickstart software envs
    • python virtual environments
    • additional software
    • community software esmvaltool
    • community software checksit
    • compiling and linking
    • conda environments and python virtual environments
    • conda removal
    • creating and using miniforge environments
    • idl
    • jasmin sci software environment
    • jasmin software faqs
    • jaspy envs
    • matplotlib
    • nag library
    • name dispersion model
    • geocat replaces ncl
    • postgres databases on request
    • running python on jasmin
    • running r on jasmin
    • rocky9 migration 2024
    • share software envs
  • data transfer
    • data transfer overview
    • data transfer tools
    • globus transfers with jasmin
    • bbcp
    • ftp and lftp
    • globus command line interface
    • globus connect personal
    • gridftp ssh auth
    • rclone
    • rsync scp sftp
    • scheduling automating transfers
    • transfers from archer2
  • short term project storage
    • apply for access to a gws
    • elastic tape command line interface hints
    • faqs storage
    • gws etiquette
    • gws scanner ui
    • gws scanner
    • gws alert system
    • install xfc client
    • xfc
    • introduction to group workspaces
    • jdma
    • managing a gws
    • secondary copy using elastic tape
    • share gws data on jasmin
    • share gws data via http
    • using the jasmin object store
    • configuring cors for object storage
  • long term archive storage
    • ceda archive
  • mass
    • external access to mass faq
    • how to apply for mass access
    • moose the mass client user guide
    • setting up your jasmin account for access to mass
  • for cloud tenants
    • introduction to the jasmin cloud
    • jasmin cloud portal
    • cluster as a service
    • cluster as a service kubernetes
    • cluster as a service identity manager
    • cluster as a service slurm
    • cluster as a service pangeo
    • cluster as a service shared storage
    • adding and removing ssh keys from an external cloud vm
    • provisioning tenancy sci vm managed cloud
    • sysadmin guidance external cloud
    • best practice
  • workflow management
    • rose cylc on jasmin
    • using cron
  • uncategorized
    • mobaxterm
    • requesting resources
    • processing requests for resources
    • acknowledging jasmin
    • approving requests for access
    • working with many linux groups
    • jasmin conditions of use
  • getting started
    • get started with jasmin
    • generate ssh key pair
    • get jasmin portal account
    • get login account
    • beginners training workshop
    • how to contact us about jasmin issues
    • jasmin status
    • jasmin training accounts
    • tips for new users
    • how to login
    • multiple account types
    • present ssh key
    • reconfirm email address
    • reset jasmin account password
    • ssh auth
    • storage
    • understanding new jasmin storage
    • update a jasmin account
  • interactive computing
    • interactive computing overview
    • check network details
    • login servers
    • login problems
    • graphical linux desktop access using nx
    • sci servers
    • tenancy sci analysis vms
    • transfer servers
    • jasmin notebooks service
    • jasmin notebooks service with gpus
    • creating a virtual environment in the notebooks service
    • project specific servers
    • dask gateway
    • access from vscode
  • batch computing
    • lotus overview
    • slurm scheduler overview
    • slurm queues
    • lotus cluster specification
    • how to monitor slurm jobs
    • how to submit a job
    • how to submit an mpi parallel job
    • example job 2 calc md5s
    • orchid gpu cluster
    • slurm status
    • slurm quick reference
  • software on jasmin
    • software overview
    • quickstart software envs
    • python virtual environments
    • additional software
    • community software esmvaltool
    • community software checksit
    • compiling and linking
    • conda environments and python virtual environments
    • conda removal
    • creating and using miniforge environments
    • idl
    • jasmin sci software environment
    • jasmin software faqs
    • jaspy envs
    • matplotlib
    • nag library
    • name dispersion model
    • geocat replaces ncl
    • postgres databases on request
    • running python on jasmin
    • running r on jasmin
    • rocky9 migration 2024
    • share software envs
  • data transfer
    • data transfer overview
    • data transfer tools
    • globus transfers with jasmin
    • bbcp
    • ftp and lftp
    • globus command line interface
    • globus connect personal
    • gridftp ssh auth
    • rclone
    • rsync scp sftp
    • scheduling automating transfers
    • transfers from archer2
  • short term project storage
    • apply for access to a gws
    • elastic tape command line interface hints
    • faqs storage
    • gws etiquette
    • gws scanner ui
    • gws scanner
    • gws alert system
    • install xfc client
    • xfc
    • introduction to group workspaces
    • jdma
    • managing a gws
    • secondary copy using elastic tape
    • share gws data on jasmin
    • share gws data via http
    • using the jasmin object store
    • configuring cors for object storage
  • long term archive storage
    • ceda archive
  • mass
    • external access to mass faq
    • how to apply for mass access
    • moose the mass client user guide
    • setting up your jasmin account for access to mass
  • for cloud tenants
    • introduction to the jasmin cloud
    • jasmin cloud portal
    • cluster as a service
    • cluster as a service kubernetes
    • cluster as a service identity manager
    • cluster as a service slurm
    • cluster as a service pangeo
    • cluster as a service shared storage
    • adding and removing ssh keys from an external cloud vm
    • provisioning tenancy sci vm managed cloud
    • sysadmin guidance external cloud
    • best practice
  • workflow management
    • rose cylc on jasmin
    • using cron
  • uncategorized
    • mobaxterm
    • requesting resources
    • processing requests for resources
    • acknowledging jasmin
    • approving requests for access
    • working with many linux groups
    • jasmin conditions of use
  1.   Short term project storage
  1. Home
  2. Docs
  3. Short term project storage
  4. Transfer Cache (XFC)

Transfer Cache (XFC)

 

Share via

Transfer Cache (XFC)

On this page
What is the XFC?   Quotas   Default Quota values:   Automatic deletion   Using XFC   Example of initial use   initial setup   query the quota  

What is the XFC?  

The Transfer Cache (XFC) provides a large area of temporary storage for users of JASMIN to store large files and/or a large volume of files on a short-term basis.

Users are granted a quota of space in their user area on the temporary storage. When users exceed their quota some of their files will be deleted automatically.

Users interact with the XFC in two ways:

  1. to initialise their user area, and to get information about their quota, a command-line client is used.
  2. to move data in and out of their user area, the standard UNIX command-line tools (cp, mkdir, rm, mv, rsync, etc.) are used.

Quotas  

XFC has two different types of quota. The first is the “hard quota” (HQ). This is the absolute maximum volume of data that can be stored in the user area. This is expressed in TB (terabytes).

The second type of quota is the “temporal quota” (TQ). This is expressed in units of TB day (terabyte days), and has a time component as well as a data volume component. For an individual file, the TQ for that file is the product of the size of the file and the number of days the file has been in the user area. As an example, if the user moves a 2TB file into their area, after 24 hours it will have used 2TB days of the TQ. After 48 hours it will have used 4TB days and after 1 week it will have used 14TB days.

Finally, any file in the XFC can have a maximum persistence of 365 days. i.e. if a file is in XFC for more than one year then it will be deleted by the automatic deletion.

image

The above figure shows an example of the quota system in use. The red line shows the temporal quota used (TQ) and the blue line shows the hard quota (HQ).

  • The user initialises their XFC.
  • On day 5 , the user copies a 1 TB file into their XFC
  • For the next 4 weeks (on days 12 , 19 , 26 and 31 ) the user copies in another 1 TB file
  • The TQ steadily grows until on day 79 it has reached its limit of 300TB days, the first 1TB file is deleted
  • On day 98 another 1TB file is deleted
  • On day 120 , the user copies 10TB into their XFC
  • On days 122 , 130 and 140 the 1TB files that were copied in on days 19 , 26 and 31 are deleted.
  • On day 151 , the 10TB file is deleted.

Default Quota values:  

The default Hard Quota (HQ) is 10 TB

The default Time Quota (TQ) is 300 TB

So you could store 10TB of data for 30 days before risking the deletion of data.

Additionally, an overall time limit of 365 days is set for ALL data stored by a given user. You cannot store any data, no matter how small, for longer than 365 days.

Automatic deletion  

If users exceed either their temporal quota or hard quota, then files in their user area will be deleted automatically. The deletion process will delete as many files as necessary to bring the amount of HQ and TQ below the quotas allocated to the user. Files will be deleted on an age basis. Those files that were copied into the user area first will be deleted first, with newer files deleted after these.

The user can be notified which files will be deleted if they switch the option to be notified on and supply an email address in the XFC client. Files will be deleted 24 hours after the notification.

If a file is modified between the notification and the scheduled deletion (24 hours later) then the file will not be deleted. However, the automatic deletion is relentless and it will choose some other file to delete instead. XFC is not designed as permanent storage and the automatic deletion process has been designed to discourage long term storage of files on it.

Using XFC  

JASMIN provides access to XFC via a command-line client: xfc

Once installed into your $HOME directory (using one of the sci servers), the xfc client can be run on either the sci (sci*.jasmin.ac.uk) or xfer (xfer*.jasmin.ac.uk) servers, but should NOT be run on the high-performance transfer servers hpxfer*.jasmin.ac.uk.

The client is used only for interacting with the service, but is not needed for accessing the storage it provides. The storage provided is mounted in most places across JASMIN: the path to your XFC volume is returned by the client in one of the steps shown below.

Users are expected to use the xfer servers or a high-performance data transfer service to do any data transfers either within or in/out of JASMIN. This reduces the load on the sci servers which are for general-purpose interactive computing.

The xfc client is used to initialise and then query the status (quota, scheduled deletions etc) of a user’s XFC storage volume:

  1. To see all the available options: xfc -h
  2. To initialise your user area: xfc init
xfc init
** SUCCESS ** - user initiliazed with:
username: username
email: user.name@stfc.ac.uk
quota: 300TB
path: /work/xfc/vol1/user_cache/username

The path is the path on the JASMIN system to the user area. Data can be copied here using standard UNIX command-line tools cp, mv, rsync. Subdirectories can be created using mkdir. Change read/write permissions on the directories and files using chmod, etc. The user area is just a standard POSIX directory and so any POSIX commands can be used on it.

  1. To get the user area path again:
xfc path
/work/xfc/vol1/user_cache/username
  1. To set the user email for notifications:
xfc email --email=user.name@stfc.ac.uk
** SUCCESS ** - user email updated to: user.name@stfc.ac.uk
  1. To query the email set for the user: xfc email
xfc email
user.name@stfc.ac.uk
  1. To switch deletion notifications on / off:
xfc notify
** SUCCESS ** - user notifcations updated to: on
  1. To see remaining quota:
xfc quota
------------------------
Quota for user: username
------------------------
Temporal Quota (TQ)
  Used : 1.7 TB
  Allocated : 300.0 TB
  Remaining : 298.3 TB
------------------------
Hard Quota (HQ)
  Used      : 444.9 GB
  Allocated : 40.0 TB
  Remaining : 39.6 TB
  1. To see which files are scheduled for deletion:
xfc schedule
No files scheduled for deletion
  1. To list the files in your user area:
xfc list
user_cache/username/historical/.ftpaccess
user_cache/username/historical/00README_catalogue_and_licence.txt
user_cache/username/historical/day/atmos/day/r1i1p1/COPY_CURRENT_20150326.txt

Pattern matching can be used to search for a file. This is just a simple substring search, e.g. r1i1p1_19500101-19541231.nc

xfc list -m r1i1p1_19500101-19541231.nc
user_cache/username/historical/day/atmos/day/r1i1p1/v20120907/va/va_day_CMCC-CESM_historical_r1i1p1_19500101-19541231.nc
user_cache/username/historical/day/atmos/day/r1i1p1/v20120907/rsds/rsds_day_CMCC-CESM_historical_r1i1p1_19500101-19541231.nc
user_cache/username/historical/day/atmos/day/r1i1p1/v20120907/prc/prc_day_CMCC-CESM_historical_r1i1p1_19500101-19541231.nc

File names are given relative to the user_cache/ directory. To list the full file path use the -f list option:

xfc list -f
  1. To predict when the files will be deleted, if no other files are added to the user area, and none of the current files are removed: xfc predict
xfc predict
Quota is predicted to be exceeded on 21 Aug 2019 14:58 by 252.1 GB
Files predicted to be deleted
user_cache/username/historical/.ftpaccess
user_cache/username/historical/00README_catalogue_and_licence.txt
user_cache/username/historical/day/atmos/day/r1i1p1/COPY_CURRENT_20150326.txt

Example of initial use  

Below is a list of commands the user might use in their initial session with XFC.

initial setup  

xfc init
xfc path
xfc email --email=user.name@email.com
xfc notify

query the quota  

xfc quota
xfc predict
xfc schedule
Last updated on 2024-10-01 as part of:  updates oct 01 for r9 env (cfbd4f95b)
On this page:
What is the XFC?   Quotas   Default Quota values:   Automatic deletion   Using XFC   Example of initial use   initial setup   query the quota  
Follow us

Social media & development

   

Useful links

  • CEDA Archive 
  • CEDA Catalogue 
  • JASMIN 
  • JASMIN Accounts Portal 
  • JASMIN Projects Portal 
  • JASMIN Cloud Portal 
  • JASMIN Notebooks Service 
  • JASMIN Community Discussions 

Contact us

  • Helpdesk
UKRI/STFC logo
UKRI/NERC logo
NCAS logo
NCEO logo
Accessibility | Terms and Conditions | Privacy and Cookies
Copyright © 2025 Science and Technology Facilities Council.
Hinode theme for Hugo licensed under Creative Commons (CC BY-NC-SA 4.0).
JASMIN Help Site
Link copied to clipboard
JASMIN Help Site
Code copied to clipboard