Using Cron
Using Cron
Cron is a very common job scheduler for linux. It allows users to run the same command or shell script periodically. Typically it is used to automate tasks, for example, every Monday run my script to plot last week’s data. There are many guides to using cron and crontab (the command for loading the cron job table).
Generally cron is disabled on the JASMIN general access machines such as
sci1.jasmin.ac.uk
. This is to avoid people killing the machine by setting up
lots of processing jobs when better alternatives, e.g. Lotus, are available.
However, there are times when it is appropriate to use cron and so a generic
cron service machine is provided. cron.jasmin.ac.uk
is configured like
sci*.jasmin.ac.uk
, except cron is enabled. Anyone who can log into sci*
should also be able to login to cron.jasmin.ac.uk
. (the actual hostname of
the machine is cron1.ceda.ac.uk
but please refer to it as
cron.jasmin.ac.uk
wherever possible).
An additional transfer server xfer3.jasmin.ac.uk
is equipped with cron
for
scheduling transfers only (no processing), although other methods for
scheduling/automating transfers are available.
There are a few rules of the road to using this service:
The bash shell environment when cron launches scripts is not identical to the one when working interactively. Annoyingly the path to common tools, like sbatch, may not have been setup so that you get an error message from cron when it works perfectly well interactively. A way to get round this is to source the .bash_profile in the crontab file so that the interactive environment is used by cron.
24 * * * * . $HOME/.bash_profile; sbatch -W 12:0 mycmd.sh
Crontamer is a wrapper script to implement lock file and time out checking for cron jobs to avoid issues of ‘’’runaway’’’ jobs causing problems for everyone (e.g. where new jobs start before old ones finish, which can cause a snowball effect).
You can download the crontamer script from
https://github.com/cedadev/crontamer. It is already installed on
cron1.ceda.ac.uk
Use cron as normal, but include the crontamer command before the users own script and arguments.
crontab -l
## e.g. cron file entry to run job every day at 4am:
0 4 * * * crontamer -t 2h '/home/users/jblogs/bin/my_repeat_script.sh -opt1 arg1 arg2'
Note: Crontamer has a number of options which can be conflated with the options for the wrapped script. Use single quotes to make sure the script runs with the right options.
The flow of the crontamer script is like this:
Unless a named lock file is given the lock files are created in the /tmp directory as
/tmp/crontamer.{unique_id}
These files should be cleaned up automatically, but users may need occasional checks or find them helpful for identifying problems running their scripts. The {unique_id} is based on a combination of the username, passed script and arguments, enabling multiple calls of a script with different arguments to be handled separately.
All the principles above apply whether using cron on:
cron.jasmin.ac.uk
(for initiating processing workflows)xfer3.jasmin.ac.uk
(for initiating automated transfers)