Using a CPU farm
Last Time and Todayy
Last time we discussed strings in the context
of PERL.
Other advanced scripting languages have
similarly powerful tools.
Today we talk about CPU farms.
farms
To maximise their power we will take advantage
of advanced scripting
scripting.
What is a CPU farm.
A CPU ffarm iis a collection
ll ti off processors th
thatt
can be used to process many jobs in parallel.
It is not strictlyy speaking
p gpparallel p
processing
g
as the jobs can be carried out in series.
CPU farms can allow you to carry out much
more intensive analysis than if possible if you
just has a single CPU.
What does a farm possess.
p
A front
f t end
d
This will be the machine that you log onto.
Disk
There should be “a lot” of disk space available that you can
access from the front end and the nodes.
Many farm nodes.
These are the CPUs that do the work.
They will often also have their own local disk.
A network
The nodes will be connected to the front end by a network.
The network capacity can be the limiting factor.
The front end
When using g the farm yyou will spend
p most of yyour
time on the front end.
Typically this will have the same OS as the nodes.
You can compile code here.
You submit jobs from the front end to the nodes.
You manage the disk on the front end
You might take a quick look at the output here
here.
Remember the front end will have many other users,
so try and be as undisruptive as possible
possible.
The nodes
The nodes are where your CPU time occurs.
Usually they will have local disk.
Using this will cut down on network traffic.
Improves farm performance
performance.
Be careful about how much space is available.
On some farms the same box may be several
nodes.
Dual CPU machines
Hyperthreading.
They will have high memory, but watch out for
programs with
ith very hi
high
h memory usage, th
they may
not play well together.
JJobs on a farm
JJobsb on a ffarm may be
b th
thought
ht off a bit lik
like
files on a file system.
There are commands that can
list them
delete them
Theyy have an owner
You cannot
copy them
do (many) operations on them.
An example.
p
To illustrate the use of a farm I will use an
example.
The CPU farm at RAL PPD
Generally you will be able to get an account
here if you need it (hep).
The commands are one implementation of
the grid engine.
Most farms use the same commands.
All farms have commands that do the same thing.
g
Submittingg a job
j
You use the command qsub
This will return a report to the screen, that
includes a unique job id for the job you just
submitted.
Y may need
You d thi
this jjob
b id llater.
t
qsub my_job.scr
Listingg the jobs
j submitted to a farm.
To list the jjobs use the command q
qstat
This will tell you
The job name
The job ID
It’s status
The running time
The owner.
Use qstat –u username to see the jobs belonging to
a particular user.
There are other useful switches
See the man page.
p g
JJob Status
Running
A job that is currently running on a node.
You will be able to see how long it’s running with qstat
Queued
A job that is waiting for a free node.
Terminated
A job that is finished. You won’t see these in qstat
Suspended/Error
Something has happened to the job and it’s in a error state.
This is probably your fault.
But it could be a system error, so it’s worth restarting these
once.
once
Deletingg a job
j
You can use the qdel command
qdel jobid
The job id can be obtained from qsub or
qstat.
t t
You will onlyy be able to delete yyour own jjobs.
Unless you have manager privileges.
Different queues
q
Some farms (like RAL) have different queues.
These are to separate
p resources for different
groups (experiments) from the main queue.
There may also be a fast queue
A queue with few nodes and a short maximum
duration
Good for testing.
The queue is specified in the qsub command.
Maximum duration
Some q
queues expect
p a maximum duration.
When exceeded the job will be terminated.
Set using qsub
At ral qsub –l cput=24:00:00 myjob.scr
The duration you set can control which queue
you use and the resources available.
Be careful when setting the maximum
duration try and keep it short
duration, short, but long
enough that your job will finish.
The local disk
Use the local disk whenever possible
Copy
y data files to it at the beginning.
g g
Use it for output and temp files.
Copy output to main disk at the end.
Take care however not to fill the disk.
O many systems
On t the
th llocall di
diskk iis available
il bl
as $TMPDIR
General advice
Use a shell script to control your job
job.
Don’t directly submit your executable.
Thi gives
This i you more control.
t l
Use an advanced script to control your queue
usage.
Write command files
Write job shell scripts.
Submit jobs
jobs.
Don’t submit too many jobs at once.
A typical
yp job
j script.
p
#!/bin/tcsh Setup my environment
source ~/env
/env_script.csh
script.csh
Use the local disk.
cd $TMPDIR
Copy needed data to local disk
cp ~/input_data/my_data .
$SNO_CODE/snoman.exe
_ –c mycmd.cmd
y
Run my analysis code
cp result.ntp ~/output_data/
Copy my results back to
the data disk
JJob Master Script
p
You
ou ca
can pu
put much
uc that
a we eddiscussed
scussed in the
e
last two lectures into action.
Writing
t g multiple
u t p e co
command
a d files
es a
and
d sshell
e scscripts.
pts
Running system programs and analysing their
output.
Examining the output of your analysis programs.
You can put limits on the number of jobs in
two ways.
Using the sleep command when too many jobs
are submitted.
Usingg a cronjob
j
Cronjobs
j
A cronjob
j b iis a jjob
b th
thatt runs att a scheduled
h d l d
time.
Your cronjobs are controlled by your crontab.
Not allowed on all systems (including RAL).
To edit your crontab use
crontab –e
You will use your $EDITOR variable to decide the
editor
You need to exit the editor for the change to take
effect.
A typical
yp contrab. Redirect output.
p
Otherwise you’ll get
Time to run job an email
# My program
0 * * * * my_program
y_p g > /dev/null 2>> /dev/null
#End of crontab.
Comment to end crontab. Need a newline
at the end of each command
Time is specified by five variables.
mhdwm
* is a wild card that means any
y
When the system time equals this time the job will
run.
The GRID
The GRID is an extension of single site
farms.
A farm of farms.
It will be used extensively in LHC and other
experiments.
Th use off this
The thi iis b
beyond
d th
the scope off thi
this
course.
When the time comes to use it you can talk to
your collaborators.
Exercises
Adapt your multiple command file script to
control job submission to a farm.
Write a program to send you a simple email
email,
and add it to your crontab so it will send it at
midnight.
midnight
