Using the Cluster Ohio Cluster at Kent and the IA32 Cluster at OSC

The Cluster Ohio Cluster at Kent is called co1.cs.kent.edu and has
Eight nodes each consisting of:

• Four, 550 Megahertz Intel Pentium III Xeon processors, with
  • 512kB of secondary cache
  • Two Gigabytes of RAM
  • 18 Gigabytes, ultra-wide SCSI hard drives
  • Two Myrinet interfaces
  • One 100Base-T Ethernet interface
• RedHat Linux
• Portland Group Compilers for FORTRAN
• KAI Toolset for C, and C++
• Etnus Totalview
• NAGlib
• Miscellaneous tools from the public domain or OSC developed.

• Getting started
  • File system
  • Executing programs
  • Batch requests
• Programming environment
  • Compiling systems
  • Shared memory
  • Message Passing Interface (MPI)
  • Debugging
  • Performance Analysis
• Software
• Training

Getting started at Kent

        co1.cs.kent.edu

Getting started at OSC

        oscbw.osc.edu

File system

The Cluster also has fast local disk space intended for temporary files. You are encouraged to perform the majority of your work in the temporary space and only store permanent files in your home directory. Large files stored in your home directory may be automatically migrated to the tape storage repository. To ensure fast access to required files, copy the files to the temporary area at the start of your session.

The following example shows how to use /tmp, the temporary-storage directory.

mkdir /tmp/$USER	Create your own temporary directory.
cp files /tmp/$USER	Copy the necessary files.
cd /tmp/$USER	Move to the directory.
...	Do work (compile, execute, etc.).
...
cp new files $HOME	Copy important new files back home.
cd $HOME	Return to your home directory.
rm -rf /tmp/$USER	Remove your temporary directory.
exit	End the session.

Use this procedure when compiling and executing interactively. The temporary space is not backed up, and old files may be purged when the temporary file system gets full.

A simpler procedure is available for batch jobs through the TMPDIR environment variable. See "Batch requests" for more information.

Executing programs

Commands on the Cluster can be executed either interactively or through batch requests. The Cluster has fixed usage limits for interactive execution; jobs that take more than the allowed CPU time must be executed using batch requests. To use the resources of the Cluster most efficiently, you are encouraged to use batch requests whenever possible. See "Batch requests" for more information.

For information on how to execute an MPI program, see the "MPI" section.

To execute a non-MPI program, simply enter the name of the executable. Unless otherwise specified, the number of processors used for a non-MPI parallel program is determined by the operating system at runtime. To control the number of processors, set the environment variables NCPUS (for automatically parallelized jobs) or OMP_NUM_THREADS (for jobs parallelized using the OpenMP compiler directives). If the number of available processors (four per node) is less than NCPUS or OMP_NUM_THREADS, then at least one processor will run multiple threads.

The following ksh example causes a.out to use 2 processors if they are available.

   export NCPUS=2
        export OMP_NUM_THREADS=2
        ./a.out

        integer function omp_num_threads

        int omp_num_threads();

Batch requests

The following options are often useful when submitting batch requests. The options may appear on the qsub command line or preceded by #PBS at the begining of the batch-request file.

Option	Meaning
-l walltime=time	Total wallclock time limit in seconds
-l nodes=numnodes:ppn=numprocs	Request use of numprocs processors (max 2) on each of numnodes nodes (max 112).
-N job	Name the job.
-S shell	Use shell rather than your default login shell to interpret the job script.
-j oe	Redirect stderr to stdout.
-m ae	Send e-mail when the job finishes or aborts.

By default, your batch jobs execute in your home directory. This is true even if you submit the job from another directory.

To facilitate the use of temporary disk space, a unique temporary directory is automatically created at the beginning of each batch job. This directory is also automatically removed at the end of the job. You access the directory through the TMPDIR environment variable. Note that in jobs using more than one node, $TMPDIR is not shared -- each node has its own distinct version of $TMPDIR.

A sample ksh request file appears below. The request first copies a Fortran file from a subdirectory of the user's home to the temporary space. It then compiles the file for automatic parallel execution, runs the executable using 2 threads on 1 node, and copies the results back to the previous subdirectory. Notice that the careful use of full file names allows this request to be submitted safely from any subdirectory.

#PBS -l walltime=1:00:00
#PBS -l nodes=1:ppn=2
#PBS -N my_job
#PBS -S /bin/ksh
#PBS -j oe

cd $TMPDIR
cp $HOME/science/my_program.f .
f77 -O2 -Mconcur my_program.f
export NCPUS=2
./a.out > my_results
cp my_results $HOME/science/.

        qsub my_request.job

Single-CPU sequential jobs should either set the -l nodes resource limit to 1:ppn=1 or leave it unset entirely. The following is an example of a sequential job which uses $TMPDIR as its working area.

#PBS -l walltime=40:00:00
#PBS -l nodes=1:ppn=1
#PBS -N cdnz3d
#PBS -j oe
#PBS -S /bin/ksh

cd $HOME/Beowulf/cdnz3d
cp cdnz3d cdin.dat acq.dat cdnz3d.in $TMPDIR
cd $TMPDIR
/usr/bin/time ./cdnz3d > cdnz3d.hist
cp cdnz3d.hist cdnz3d.out $HOME/Beowulf/cdnz3d

All batch jobs must set the -l walltime resource limit, as this allows the Maui Scheduler to backfill small, short running jobs in front of larger, longer running jobs. This in turn helps improve turnaround time for all jobs.

Programming environment

Compiling systems

The following examples produce the Linux executable a.out for each type of source file.

FORTRAN 77	pgf77 sample.f
Fortran 90	pgf90 sample.f90
HPF	pghpf sample.hpf
C	pgcc sample.c
C++	pgCC sample.C

For more information on command-line options for each compiling system, see the man pages (man pgf90, man pgcc, man pgCC, etc.).

Shared memory

        pgf77 -Mconcur sample.f

        pgf90 -Mconcur sample.f90

        pgcc -Mconcur sample.c

    pgCC -Mconcur sample.C

Message Passing Interface (MPI)

Each program file using MPI must include the MPI header file. The following statement must appear near the beginning of each C file or Fortran file, respectively.

        #include <mpi.h>

        include 'mpif.h'

To compile an MPI program, use the MPI wrappers around the Portland Group compilers. Here are some examples:

        mpif77 sample.f

        mpif90 sample.f90

        mpicc sample.c

        mpiCC sample.C

        pghpf -Mmpi sample.hpf

        mpiexec a.out

#PBS -l walltime=1:00:00
#PBS -l nodes=64:ppn=2
#PBS -N my_job

#PBS -S /bin/ksh
#PBS -j oe

cd $HOME/science
mpif77 -fast mpiprogram.f
pbsdcp a.out $TMPDIR
cd $TMPDIR
mpiexec ./a.out > my_results
cp my_results $HOME/science/.

mpiexec will normally spawn one MPI process per CPU requested in a batch job. However, this behavior may be modified with the -pernode command line options. The -pernode option requests that one MPI process be spawned per node. These options are intended to be used for codes which mix MPI message passing with some form of shared memory programming model, such as OpenMP or POSIX threads.

The pbsdcp command used in the example above is a distributed copy command; it copies the listed file or files to the specified destination (the last argument) on each node of the cluster assigned to your job. This is needed when copying files to directories which are not shared between nodes, such as /tmp or $TMPDIR.

Debugging

The Portland Group compiler suite includes a graphical debugger called Xpgdbg. This debugger allows for interactive and post-mortem analysis of sequential and shared-memory parallel programs. To debug a program with Xpgdbg, first compile the program with the -g option.

        pgf77 -g program.f

        pgf90 -g program.f90

        pgcc -g program.c

        pgCC -g program.C

       Xpgdbg a.out

       Xpgdbg a.out core

Performance Analysis

Software

Training