User Guide

This guide provides a brief reference for users to start using Pegasus 2.

User Environment

Users with active accounts may access Pegasus 2 by opening an SSH connection to pegasus2.ccs.miami.edu. By default, almost all Linux and Mac machines will have an SSH client. Windows users will require a terminal emulator like Putty.

For all data transfers, please use pegasus2-gw.ccs.miami.edu. Pegasus2-gw is a dedicated gateway system that is optimized for data transfers. Secure copy (SCP) and secure FTP (SFTP) are the preferred means to transfer data between CCS and local machines. For users who prefer a graphical client we recommend FileZilla.

The operating system on Pegasus 2 is CentOS 6.2. Bash is the default shell. Contact user support or log into CCS account to change your default shell.

All active users are allocated a logical storage area, HOME directory, which is limited to 250G. Home directories are intended primarily for basic account information, source codes, and binaries. There is another directory, named “/scratch”, which should be used for compiles and run-time input and output files. Both HOME and scratch are available on all nodes of the cluster. Data in the “/scratch” directory are NOT backed up and untouched data older than 3 weeks are subject to purging.

Using Modules

CCS continually updates applications, compilers, system libraries, etc. To facilitate this task and to provide a uniform mechanism for accessing different revisions of software, CCS uses the modules utility.

At login, modules commands set up a basic environment for the default compilers, tools, and libraries. For example: the $PATH, $MANPATH, $LIBPATH environment variables. Therefore, there is no need for you to set them or update them when updates are made to system and application software.

The table below lists commonly used modules commands.

Command Purpose
module avail lists all available modules
module load package loads a module e.g., the ifort package
module list list modules currently loaded
module switch old new module switch switch intel gcc replace old module with new
module purge restores original setting by unloading all modules

You may request new software for Pegasus2 using our Software Request form.

Application Development

Compiling Code

The following sections present the compiler invocation for serial and MPI executions. All compiler commands can be used for just compiling with the -c option (create just the “.o” object files) or compiling and linking (to create executables). To use a different (non-default) compiler you first unload intel, swap the compiler environment, and then reload the MPI environment if necessary.

Compiling Serial Code

Pegasus 2 has intel and gcc compilers

Vendor Compiler Module Command Example
intel icc (default) module load intel icc -o foo.exe foo.c
intel ifor (default) module load intel ifort -o foo.exe foo.f90
gnu gcc module load gcc gcc -o foo.exe foo.c
gnu gcc module load gcc gfortran -o foo.exe foo.f90

Compiling Programs with MPI

Pegasus 2 supports Intel MPI and OpenMPI and mvapich2 for the intel and gcc compilers.

Compiler MPI Module Command Example
intel Intel MPI module load intel impi mpif90 -o foo.exe foo.f90
intel Intel MPI module load intel impi mpicc -o foo.exe foo.c
intel OpenMPI module load intel openmpi mpif90 -o foo.exe foo.f90
intel OpenMPI module load intel openmpi mpicc -o foo.exe foo.c
intel mvapich2 module load intel mvapich2 mpif90 -o foo.exe foo.f90
intel mvapich2 module load intel mvapich2 mpicc -o foo.exe foo.c
gcc OpenMPI module load openmpi-gcc mpif90 -o foo.exe foo.f90
gcc OpenMPI module load openmpi-gcc mpicc -o foo.exe foo.c
gcc mvapich2 module load mvapich2-gcc mpif90 -o foo.exe foo.f90
gcc mvapich2 module load mvapich2-gcc mpicc -o foo.exe foo.c

Transferring Files to Pegasus 2 from remote host

Pegasus supports multiple file transfer programs, common command line utilities such as scp, scp, and rsync, When executing multiple instantiations of scp or rsync, please limit your transfers to no more than 2-3 processes at a time. Command-line Data Transfer SSH Data transfer from any Unix/Linux system can be accomplished using the scp utility to copy data to and from the login nodes.

Note that you may pull data from remote sites to the login nodes, however, pushing data directly to login nodes is not allowed. The copy destination must be pegasus2-gw.ccs.miami.edu.

A file can be copied from your local system to the remote server using rsync or scp. An example transfer might look like this:

login1% scp filename \
            username@pegasus2-gw.ccs.miami.edu:/scratch/directory

Consult the man pages for more information on scp.

The rsync command is another way to keep your data up to date. In contrast to scp, rsync transfers only the actual changed parts of a file (instead of transferring an entire file). Hence, this selective method of data transfer can be much more efficient than scp. The following example demonstrates usage of the rsync command for transferring a file named “firstExample.c” from the current location to pegasus.

login1$ rsync firstExample.c \
            username@ pegasus2-gw.ccs.miami.edu:/scratch/data

An entire directory can be transferred from source to destination by using rsync as well. For directory transfers the options -atvr will transfer the files recursively (-r option) along with the modification times (-t option) and in the archive mode (-a option)

Running Applications

Pegasus is currently running the LSF resource manager to schedule all compute resources. You may compile and test jobs on login nodes. However, any jobs exceeding one hour of run time or using excessive resources on the login nodes will be terminated and user’s account may be suspended.

LSF Batch System

Common LSF commands and description

Command Purpose
bsub < ScriptFile Submits a job via script file to LSF to be run. NOTE: the redirection symbol, “<”, is a must when submitting the job
bjobs Displays running and pending jobs in the queue.
bhist Displays historical information about your finished jobs.
bkill Removes/cancels a job or jobs from the class.
bqueues Shows the current configuration of queues.
bhosts Shows the load on each node.
bpeek Displays stderr and stdout from your unfinished job.

Example Usage:

The command bsub < ScriptFile will submit the given script for processing. You must write a script containing the information LSF needs to allocate the resources your job requires, to handle standard I/O streams, and to run the job. Please see the example scripts below. On submission, LSF will return the job id.

[user@login3]:>bsub < test.job
Job <4225> is submitted to general queue .

The commands bjobs will show all jobs currently running or queued on the system.

[user@login3]$ bjobs
JOBID  USER   STAT  QUEUE    FROM_HOST  EXEC_HOST   JOB_NAME  SUBMIT_TIME
4225   user   RUN   general  m1         8*n0060     testjob   Mar  2 11:53
                                        8*n0061
                                        8*n0063
                                        8*n0064
For details about your particular job, issue the command bjobs -l JobID where JobID is obtained from the "JOBID" field of the above bjobs output.
 The command bkill JobID will remove the job from the quue and terminate the job if it is running.

[user@login3]$ bkill 4225
Job <4225> is being terminated

Example script for a serial Job

#!/bin/bash
#BSUB -J serialjob
#BSUB -o %J.out
#BSUB -e %J.err
#BSUB -W 1:00
#BSUB -q general
#BSUB -n 1
#BSUB -B
#BSUB -N
#
# Run serial executable on 1 cpu of one node
# cd /nethome/jbaringer/example
cd ${HOME}/path/to/current/directory
./test.x a b c

Here is a detailed line-by-line breakdown of the keywords and their assigned values listed in this script:

#!/bin/bash
Specifies the shell to be used when executing the command portion of the script.
The default is Bash shell.

BSUB -J serialjob
assigns a name to job. The name of the job will show in the bjobs output.

#BSUB -o %J.out
redirect std output to a specified file. In this example, %J is the JobID.

#BSUB -e %J.err
redirect std error to a specified file

#BSUB -W 1:00
set wallclock time limit of 1 hour

#BSUB -q general
specify queue to be used

#BSUB -n 1
specify number of processors. For serial job, it would be 1.

#BSUB -B
Send email at job start

#BSUB -N
Send email at job end

Example script for Intel/Intel MPI

Note that if you are porting your script from pegasus1 there are some changes:

  1. remove -a options
  2. run mpirun.hydra instead of mpirun.lsf
#!/bin/bash
#BSUB -J mpijob
#BSUB -o %J.out
#BSUB -e %J.err
#BSUB -W 1:30
#BSUB -q general
#BSUB -n 32
#
# Run an MPI job with the "mpirun.hydra" MPI job starter.
mpirun.hydra foo.exe

Example script for OpenMPI

Note that if you are porting your script from pegasus1 there are some changes:

  1. remove -a openmpi option

#!/bin/bash
#BSUB -J mpijob
#BSUB -o %J.out
#BSUB -e %J.err
#BSUB -W 1:30
#BSUB -q general
#BSUB -n 32
#
# Run an MPI job with the “mpirun.lsf” MPI job starter.
mpiexec –mca btl self,sm,openib foo.exe

Running An Interactive Job

HPC clusters primarily take batch jobs and run them in the “background”; users do not need to interact with the job during the execution.  However, sometimes users do need to interact with the application, for example, the application needs the input from the command line or waits for a mouse event in xwindows.

There are a few nodes on Pegasus2 dedicated for interactive work. These types of jobs may be launched from pegasus2-gw.ccs.miami.edu.

Use the bsub options -Is for an interactive and, additionally -XF for an X11 job like:

[warner@login3 ~]$ bsub -q interactive -Is -XF matlab
Job <50274> is submitted to queue <interactive>.
<<ssh X11 forwarding job>>
<<Waiting for dispatch …>>
<<Starting on n003.pegasus.edu>>
[warner@login3 ~]$

When you are finished using the node exit out of it and you will be returned to one of the login nodes.

Please note again interactive jobs can only be launched pegasus2-gw.ccs.miami.edu.  Additionally,  if you are running an X11 application you will need to establish and X tunnel with ssh when connecting to pegagus2-gw and use the -XF option for on bsub.  For example,

ssh -X user@pegasus2-gw.ccs.miami.edu