2017-02-17 CCS Advanced Computing – Slack community

CCS Advanced Computing invites you to connect with the Advanced Computing community on Slack: http://umadvancedcomputing.slack.com
The Advanced Computing community Slack channels provide a place for user discussions, information sharing, and informal announcements about CCS resources and developments. All users with an @miami.edu or @umiami.edu email address can create a Slack account in UM Advanced Computing.

2017-01-18 CCS Portal – portal.ccs.miami.edu

CCS Advanced Computing is excited to announce the launch of our newly redesigned Portal: http://portal.ccs.miami.edu
The new CCS Portal home page showcases up-to-date information about our clusters. Click the title for more details.

2016-09-29 Pegasus /scratch purge policy violations

Files on /scratch are subject to purging after 21 days. Circumventing this purge is a violation of University of Miami Acceptable Use Policies and the account responsible for it will be suspended. Using ‘touch’ to thwart /scratch purge is not allowed.

2016-04-15 Parallel Queue Debut

CCS AC debuts the new ‘parallel’ Pegasus cluster job queue for parallel jobs requiring 16 cores from the same host, or MPI jobs requiring 16+ CPUs.
Click title for details.

Documentation & User Guides

Advanced Computing Documentation & User Guides contains detailed information about our systems. User guides for the Pegasus Supercomputer — including high-performance clusters, software, and job submissions — and the Nyx Cloud are grouped by category. Access specific topics by clicking the arrows next to menu items and search all documentation pages with the search bar.

For help using Linux, see our Pegasus Linux User Guides.

  1. Pegasus Supercomputer
    1. Pegasus Environment
      1. File Systems
      2. Resource Allocations
      3. Job Submissions
    2. Connecting to Pegasus with SSH
      1. Windows
      2. Mac and Linux
      3. Forwarding the Display
      4. Connecting from offsite
    3. Transferring Files to Pegasus
      1. FileZilla
      2. Command Line Utilities
    4. Projects on Pegasus
      1. Joining a project
      2. Using scratch space
      3. Using projects in jobs
      4. Project Management
        1. Creating a new project
        2. Converting a group
        3. Updating a project
        4. Updating project members
  2. Software on Pegasus
    1. Application Development
      1. Compiling Serial Code
      2. Configuring MPI
      3. Compiling Parallel Code
    2. Parallel Program Examples
      1. C++
      2. C
      3. Fortran 90
      4. Fortran 77
      5. MPI LSF Job Script
    3. Installing Software on Pegasus
      1. Downloading and Extracting Files
        1. Pre-compiled software (binaries)
        2. Source Code
      2. Configuring & Compiling
      3. Updating PATH
      4. Creating symbolic links
      5. Persistent PATH
    4. Using Python Modules on Pegasus
      1. Loading and Switching
      2. Package Managers
      3. Downloaded Modules
      4. Checking Module Versions
      5. Python Virtual Environments
        1. Creating VEs
        2. Activating VEs
        3. Installing Modules in VEs
        4. Deactivating VEs
        5. Comparing VEs with diff
        6. Recreating VEs
        7. Using the Python VE Wrapper
    5. Using Perl Modules on Pegasus
      1. Configuring a Local Library
        1. CPAN
        2. local::lib Module
      2. Installing Perl Modules
        1. CPAN
        2. cpanminus
      3. Deactivating Local Libs
    6. Running Matlab on Pegasus
      1. Interactive
        1. Graphical Interface Mode
        2. Interactive -nodesktop
      2. Batch Processing
      3. Parallel Processing
        1. Single-node
        2. Multi-node
      4. Cluster Profiles
    7. Running R on Pegasus
    8. Running SAS on Pegasus
    9. SVN - Subversion
      1. SVN Server
      2. SVN Client
      3. Basic SVN Commands
      4. Basic SVN Usage
    10. Allinea - debugging and profiling
  3. Scheduling Jobs on Pegasus
    1. Submitting Batch Jobs
    2. Pegasus Job Queues
    3. LSF Commands
      1. bsub - Submit Jobs
      2. bjobs - Display Job Info
      3. bhist - Display Job History
      4. bkill - Terminate Jobs
      5. bqueues - Display Queue Info
      6. bhosts - Display Host Info
      7. bpeek - Display Running Job Info
      8. output and error files
    4. LSF Job Scripts
      1. Serial
      2. Parallel
        1. Parallel, Intel
        2. Parallel, openMPI
    5. LSF Interactive Jobs
      1. LSF Interactive Display Forwarding
  4. Nyx Cloud
    1. Nyx Cloud Environment
      1. Account requests
      2. Available Images
    2. Nyx Cloud Usage
      1. Creating SSH keys
      2. Creating VMs
      3. Controlling VMs
      4. Enabling SSH
      5. Accessing via SSH
      6. Controlling ports

Pegasus Supercomputer

Information:          Pegasus Supercomputer
Credentials:          CCS Account
Access & Allocations: Policies 
Operating System:     CentOS 6.5 
Default Shell:        Bash
Data Transfer:        SCP and SFTP

Pegasus Environment

Users with active CCS accounts may access Pegasus by opening an SSH connection to pegasus.ccs.miami.edu. Please see our Policies page for more information about CCS accounts, Pegasus projects, and allocations. Almost all Linux and Mac machines will already have an SSH client. Windows users will require a terminal emulator like PuTTY.

Secure copy (SCP) and secure FTP (SFTP) are the preferred means to transfer data between CCS and local machines. For users who prefer a graphical client, we recommend FileZilla.

Information about your account, including CPU hours used, is available when you log into Pegasus.

File Systems

Pegasus has two parallel file systems available to users:

/nethome
(permanent, quota’d, not backed-up)
/scratch
(high-speed purged storage)

/nethome directories are limited to 250GB and intended primarily for basic account information, source codes, and binaries.
/scratch directories should be used for compiles and run-time input & output files.
Stage data for running jobs exclusively in the scratch or scratch/projects filesystems.
Both home and scratch are available on all nodes in the Pegasus cluster.

No backups are performed on Pegasus file systems. Files on /scratch/ are subject to purging after 21 days.

Resource Allocations

Pegasus cluster resources are assigned and managed on a project basis for collaborative work. This method allows us to better support interaction between teams (including data sharing) at the University regardless of group, school or campus. A project is defined as a special user group created on the Pegasus cluster for a specific research project, grant, or goal. Cluster resources, including CPU hours and scratch space, are assigned to projects. See Projects (link) for more information.

Job Submission

ALWAYS submit your production jobs to the cluster using LSF, and stage data in the /scratch or /scratch/project filesystems. DO NOT run production jobs on the head nodes. To learn how to use LSF, see our Scheduling Jobs on Pegasus section (link). You may compile and test jobs on login nodes. However, any jobs exceeding 30 minutes of run time or using excessive resources on the login nodes will be terminated and the CCS account responsible for those jobs may be suspended.

Back to Top

Connecting to Pegasus with SSH

Use a secure-shell (SSH) client to connect to Pegasus for secure, encrypted communication. From within UM’s secure network (SecureCanes or wired connection on campus) or VPN, connect to Pegasus with:

Windows

Connect using a terminal emulator like PuTTY (www.putty.org)

Log in with your CCS Account credentials:

username@pegasus.ccs.miami.edu (optional username @ host)
22 (port)
SSH (connection type)

PuTTY in Windows

Mac and Linux

Connect with the Terminal program, included with the Operating Systems.

Log in with your CCS Account credentials directly:

bash-4.1$ ssh username@pegasus.ccs.miami.edu
username@pegasus.ccs.miami.edu’s password:

or be prompted for CCS Account credentials:

bash-4.1$ ssh pegasus.ccs.miami.edu
login as: username
username@pegasus.ccs.miami.edu’s password:

To use SSH key pairs to authenticate, see the CentOS wiki: http://wiki.centos.org/HowTos/Network/SecuringSSH

Pegasus Welcome Message

The Pegasus welcome message includes links to Pegasus documentation and information about your disk quotas, jobs, and cpu hours.

------------------------------------------------------------------------------
                     Welcome to the Pegasus Supercomputer
           Center for Computational Science, University of Miami 
------------------------------------------------------------------------------
...
...
...
--------------------------Disk Quota------------------------------
filesystem | project    | used(GB)   | quota (GB) | Util(%)   
============================================================
nethome    | user       | 0.76       | 250.00     |    0%
scratch    | projectID  | 93.32      | 20000.00   |    0%
------------------------------------------------------------------
     Files on /scratch are subject to purging after 21 days       
------------------------------------------------------------------
jobs  finished: 9
cpu hours used: 0
------------------------------------------------------------------
[username@pegasus ~]$

Forwarding the display

To use graphical programs on Pegasus, the graphical display must be forwarded securely. This typically requires running an X Window System server and adding the -X option when connecting via SSH.

Download an X Window System server

Connect to Pegasus with X11 forwarding

Launch the appropriate X Window server before connecting to Pegasus via SSH.

Windows: Configure PuTTY for X11 display forwarding

In PuTTY Configuration,

  • scroll to the Connection category and expand it
  • scroll to the SSH sub-category and expand it
  • click on the X11 sub-category

On the X11 Options Panel,

  • check “Enable X11 forwarding”
  • enter “localhost:0” in the “X display location” text field

PuTTY X11

Mac: Connect with X11 flag

Using either the Mac Terminal or the xterm window, connect using the -X flag:

bash-4.1$ ssh -X username@pegasus.ccs.miami.edu

Launch a graphical application

Use & after the command to run the application in the background, allowing continued use of the terminal.

[username@pegasus ~]$ firefox &

Connecting to Pegasus from offsite

Pegasus and other CCS resources are only available from within the University’s secure campus networks (wired or SecureCanes wireless). To access Pegasus while offsite, open a VPN connection first. CCS does not administer VPN accounts.

University of Miami VPN: https://www.it.miami.edu/a-z-listing/virtual-private-network/index.html

Back to Top

Transferring Files to Pegasus

Pegasus supports multiple file transfer programs such as FileZilla and PSFTP, and common command line utilities such as scp and rsync.

Using FileZilla

FileZilla is a free, user friendly, open source, cross-platform FTP, SFTP and FTPS application.

Download the FileZilla client here: https://filezilla-project.org/download.php?show_all=1 and follow the installation instructions for your platform (http://wiki.filezilla-project.org/Client_Installation).

Launch FileZilla and open File : Site Manager.

Click the “New Site” button and name your entry.

  • Host: pegasus.ccs.miami.edu
  • Protocol: SFTP
  • Logon Type: Normal
  • enter your username and password

You can also select Logon Type: Ask for password for a password prompt each connection.FileZilla Site Manager

Click the “Connect” button. Once connected, you can drag and drop files or directories between your local machine and the server.

Using command line utilities

When executing multiple instantiations of scp or rsync, please limit your transfers to no more than 2-3 processes at a time. Command-line Data Transfer / SSH Data transfer from any Unix/Linux system can be accomplished using the scp utility. A file can be copied from your local system to the remote server using rsync or scp.

scp

An example transfer might look like this:

[localmachine: ~]$ scp /local/filename \
            username@pegasus.ccs.miami.edu:/scratch/projectID/directory

To transfer a directory, use the -r flag (recursive):

[localmachine: ~]$ scp -r /local/directory \
            username@pegasus.ccs.miami.edu:/scratch/projectID/directory

Consult the man pages for more information on scp.

rsync

The rsync command is another way to keep your data up to date. In contrast to scp, rsync transfers only the actual changed parts of a file (instead of transferring an entire file). Hence, this selective method of data transfer can be much more efficient than scp. The following example demonstrates usage of the rsync command for transferring a file named “firstExample.c” from the current location to Pegasus.

[localmachine: ~]$ rsync firstExample.c \
            username@pegasus.ccs.miami.edu:/scratch/projectID/directory

An entire directory can be transferred from source to destination by using rsync as well. For directory transfers, the options -atvr will transfer the files recursively (-r option) along with the modification times (-t option) and in the archive mode (-a option).

Back to Top

Projects on Pegasus

Pegasus cluster resources are assigned and managed on a project basis for collaborative work. This method allows us to better support interaction between teams (including data sharing) at the University regardless of group, school or campus. Project-based resource allocation also gives researchers the ability to request resources for short-term work. A project is defined as a special user group created on the Pegasus cluster for a specific research project, grant, or goal. Cluster resources, including CPU hours and scratch space, are assigned to projects.

Users without projects start with the shared default project. This default project is assigned limited cluster resources (256 cores). While the default project has more than adequate resources for casual use, we strongly recommend that all PIs and faculty members create a project (or projects) for collaborative work with high resource requirements.

Joining an existing project

To review the full list of projects currently registered on the Pegasus cluster, click on List under Projects on the Forms and Access section of the CCS Portal. Click on a project name to access the membership request form.
Users can also access the membership request form directly: http://portal.ccs.miami.edu/portal/project/join/projectname

Using project scratch space

Project scratch space is located under /scratch/projects/PROJECT_NAME. Project members have read and write access to this space. All user data under project space share a common quota assigned to the project. Projects typically have larger quotas than individual requests.

Using projects in computing jobs

In order to run jobs with your project, submit jobs with your assigned projectID using the -P argument to bsub: bsub -P projectID. For more information about LSF and job scheduling, see Scheduling Jobs on Pegasus (link).

For example, if you were assigned the project id “abc”, a batch submission from the command line would look like:

$ bsub -P abc < JOB_SCRIPT_NAME

and an interactive submission from the command line would look like:

$ bsub -P abc -Is -q interactive -XF command

When your job has been submitted successfully, the project and queue information will be printed on the screen.

Job is submitted to <abc> project.

Job <11234> is submitted to default queue <general>.

If that project id does not exist, or the user is not a member of a project, the job will be submitted to the ‘default’ project.

Job is submitted to <default> project.

Job <11234> is submitted to default queue <general>.

Project Management

Project allocations are granted by fiscal year and divided into different classes with different justification requirements. For projects requesting up to 50,000 CPU hours and up to 2TB scratch space, basic information about the new project is required such as computing usage, scientific justification, and scratch usage. Additional justifications are required for projects with large resource requirements.

CCS strongly encourages users to apply for allocations above the minimum levels that reflect a realistic expectation of computations required for a given project. The allocation data are strictly for tracking purposes, not for determining appropriateness of requests. CCS will make every effort to accommodate all users of the Pegasus system.

Projects are managed through the CCS Portal, accessible from secure campus networks only.

Creating a new project

Any PI or faculty member may request a new project by clicking the Projects link on the CCS Portal (http://portal.ccs.miami.edu/portal). Once your project has been approved and created, you will be sent an email with your new project id and your resource specifics.
See our Policies page for more information.

Converting a Linux group to a new project

All group leaders can convert an existing group to a project by clicking on the ‘Convert’ link associated with a group in the My Pegasus section on the CCS Portal. Similar to creating a new project, if a project exceeds the standard quota (100,000 CPU hours) the CCS allocations committee will review the request. By converting an existing group to a project, group members become project members automatically.

Updating an existing project

If a project needs more resources than originally requested, the project leader can update the resource requirements on the ‘Update Resource’ link associated with a project on My Pegasus. Once submitted, the new CPU hour and/or scratch space size request will be sent to the CCS allocations committee for disposition. Once a decision has been made made, the requestor will receive an email notification with the update.

Only a project leader may request more resources. Project members should ask their project leaders to make requests for more resources.

Updating project members

The project leader manages all project membership. Project leaders can manage their projects on the Projects page under My Pegasus tab on the CCS Portal. Click the Edit button next to Members, type the member’s username in the field, click Add then Save.

Project members must belong to at least one group. If a project member needs to be removed from all projects (in case of graduation, separation, etc.), please contact hpc@ccs.miami.edu for assistance.

An inactive user cannot be added to a project. The user must request re-activation from hpc@ccs.miami.edu first.

Back to Top

Software on the Pegasus Cluster

CCS continually updates applications, compilers, system libraries, etc. To facilitate this task and to provide a uniform mechanism for accessing different revisions of software, CCS uses the modules utility. At login, modules commands set up a basic environment for the default compilers, tools, and libraries such as:  the $PATH, $MANPATH, and $LD_LIBRARY_PATH environment variables. There is no need to set them or update them when updates are made to system and application software. Available software modules can be viewed on the CCS Portal Software Modules page including description, version, and update date.

The table below lists commonly used modules commands.

Command Purpose
module avail lists all available modules
module list list modules currently loaded
module purge restores original setting by unloading all modules
module load package loads a module e.g., the python package
module unload package unloads a module e.g., the python package
module switch old new replaces old module with new module
module display package displays location and library information about a module

Request new global software for Pegasus using our Software Request form. See our Policies page for minimum requirements and more information.

Back to Top

Application Development

A list of software modules installed on Pegasus, including MP libraries, is available on the CCS Portal. MPI and OpenMP modules are listed under Intel and GCC compilers. These MP libraries have been compiled and built with either the Intel compiler suite or the GNU compiler suite.

The following sections present the compiler invocation for serial and MP executions. All compiler commands can be used for just compiling with the -c flag (to create just the “.o” object files) or compiling and linking (to create executables). To use a different (non-default) compiler, first unload intel, swap the compiler environment, and then reload the MP environment if necessary. Only one MP module should be loaded at a time.

Compiling Serial Code

Pegasus has Intel and GCC compilers.

Vendor Compiler Module Command Example
intel icc (default) module load intel icc -o foo.exe foo.c
intel ifor (default) module load intel ifort -o foo.exe foo.f90
gnu gcc module load gcc gcc -o foo.exe foo.c
gnu gcc module load gcc gfortran -o foo.exe foo.f90

Configuring MPI on Pegasus

There are three ways to configure MPI on Pegasus. Choose the option that works best for your job requirements.

  • Add the module load command to your startup files.
    This is most convenient for users requiring only a single version of MPI. This method works with all MPI modules.
  • Load the module in your current shell.
    For current MPI versions, the module load command does not need to be in your startup files. Upon job submission, the remote processes will inherit the submission shell environment and use the proper MPI library. This method does not work with older versions of MPI.
  • Load the module in your job script.
    This is most convenient for users requiring different versions of MPI for different jobs. Ensure your script can execute the module command properly. For job script information, see Scheduling Jobs on Pegasus (direct link).

Compiling Parallel Programs with MPI

Pegasus supports Intel MPI and OpenMP for Intel and GCC compilers.

The Message Passing Interface (MPI) library allows processes in parallel application to communicate with one another. There is no default MPI library in your Pegasus environment. Choose the desired MPI implementation for your applications by loading an appropriate MPI module. Recall that only one MPI module should be loaded at a time.

Compiler MPI Module Command Example
intel Intel MPI module load intel impi mpif90 -o foo.exe foo.f90
intel Intel MPI module load intel impi mpicc -o foo.exe foo.c
intel OpenMP module load intel openmpi mpif90 -o foo.exe foo.f90
intel OpenMP module load intel openmpi mpicc -o foo.exe foo.c
gcc OpenMP module load openmpi-gcc mpif90 -o foo.exe foo.f90
gcc OpenMP module load openmpi-gcc mpicc -o foo.exe foo.c

Back to Top

Parallel Program Examples

We recommend using Intel MPI unless you have specific reason for using OpenMP.  Intel MPI scales better and has better performance than OpenMP.

Only one MPI module should be loaded at a time.

C++ program and compilation:
//=================================================================
// C++ example: MPI Example 1
//=================================================================
#include <iostream>
#include <mpi.h>
using namespace std;
int main(int argc, char** argv){
  int iproc;
  MPI_Comm icomm;
  int nproc;
  int i;
  MPI_Init(&argc,&argv);
  icomm = MPI_COMM_WORLD;
  MPI_Comm_rank(icomm,&iproc);
  MPI_Comm_size(icomm,&nproc);
  for ( i = 0; i <= nproc - 1; i++ ){
    MPI_Barrier(icomm);
    if ( i == iproc ){
      cout << "Rank " << iproc << " out of " << nproc << endl;
    }
  }
  MPI_Finalize();
  return 0;
}
[username@pegasus ~]$ mpicxx -o mpi_example1.x mpi_example1.cpp

C program and compilation:
//=================================================================
// C example: MPI Example 1
//=================================================================
#include <stdio.h>
#include <mpi.h>
int main(int argc, char** argv){
int iproc;
int icomm;
int nproc;
int i;
MPI_Init(&argc,&argv);
icomm = MPI_COMM_WORLD;
MPI_Comm_rank(icomm,&iproc);
MPI_Comm_size(icomm,&nproc);
for ( i = 0; i <= nproc - 1; i++ ){
  MPI_Barrier(icomm);
  if ( i == iproc ){
    printf("%s %d %s %d \n","Rank",iproc,"out of",nproc);
  }
}
MPI_Finalize();
return 0;
}
[username@pegasus ~]$ mpicc -o mpi_example1.x mpi_example1.c

Fortran 90 program and compilation:
!=====================================================
! Fortran 90 example: MPI test
!=====================================================
program mpiexample1
implicit none
include 'mpif.h'
integer(4) :: ierr
integer(4) :: iproc
integer(4) :: nproc
integer(4) :: icomm
integer(4) :: i
call MPI_INIT(ierr)
icomm = MPI_COMM_WORLD
call MPI_COMM_SIZE(icomm,nproc,ierr)
call MPI_COMM_RANK(icomm,iproc,ierr)
do i = 0, nproc-1
  call MPI_BARRIER(icomm,ierr)
  if ( iproc == i ) then
    write (6,*) "Rank",iproc,"out of",nproc
  end if
end do
call MPI_FINALIZE(ierr)
if ( iproc == 0 ) write(6,*)'End of program.'
  stop
end program mpiexample1
[username@pegasus ~]$ mpif90 -o mpi_example1.x mpi_example1.f90

Fortran 77 program and compilation:
c=====================================================
c Fortran 77 example: MPI Example 1
c=====================================================
program mpitest
implicit none
include 'mpif.h'
integer(4) :: ierr
integer(4) :: iproc
integer(4) :: nproc
integer(4) :: icomm
integer(4) :: i
call MPI_INIT(ierr)
icomm = MPI_COMM_WORLD
call MPI_COMM_SIZE(icomm,nproc,ierr)
call MPI_COMM_RANK(icomm,iproc,ierr)
do i = 0, nproc-1
  call MPI_BARRIER(icomm,ierr)
  if ( iproc == i ) then
    write (6,*) "Rank",iproc,"out of",nproc
  end if
end do
call MPI_FINALIZE(ierr)
if ( iproc == 0 ) write(6,*)'End of program.'
  stop
end
[username@pegasus ~]$ mpif77 -o mpi_example1.x mpi_example1.f

LSF script example

This batch script mpi_example1.job instructs LSF to reserve computational resources for your job. Change the -P flag argument to your project before running.

mpi_example1.job

#!/bin/sh #BSUB -n 16 #BSUB -J test #BSUB -o test.out #BSUB -e test.err #BSUB -a openmpi #BSUB -R "span[ptile=16]" #BSUB -q parallel #BSUB -P hpc mpirun.lsf ./mpi_example1.x

Submit this scriptfile using bsub. For job script information, see Scheduling Jobs on Pegasus (link).

[username@pegasus ~]$ bsub -q parallel < mpi_example1.job
Job <6021006> is submitted to queue <parallel>.
...

Back to Top

Installing Software on Pegasus

Pegasus users are free to compile and install software in their own home directories by following source code installation instructions. We suggest keeping downloaded source files separate from compiled files and downloaded binaries. In general, software installation involves:

  1. downloading and extracting files
  2. configuring the installation and compiling the software (compile only)
  3. updating PATH and creating symbolic links

Confirm that your software’s pre-requisites are met, either in your local environment or on Pegasus as a module. You will need to load any Pegasus modules that are pre-requisites and install locally any other pre-requisites. CCS does not install user software. Request cluster software installations or updates via the Software Request form.

Downloading and extracting files

If necessary, create software directories under your home directory:

[username@pegasus ~]$ mkdir ~/local ~/src

We suggest keeping your compiled software separate from any downloaded files. Consider keeping downloaded binaries (pre-compiled software) separate from source files if you will be installing many different programs. These directories do not need to be named exactly as shown above.

Navigate to the src directory and download files:

Some programs require configuration and compilation (like autoconf). Other programs are pre-compiled and simply need to be extracted (like Firefox). Read and follow all instructions provided for each program.

[username@pegasus ~]$ cd ~/src
[username@pegasus src]$ wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz
[username@pegasus src]$ wget http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/36.0/linux-x86_64/en-GB/firefox-36.0.tar.bz2

Pre-compiled software can be extracted and immediately moved into your local software directory. We suggest maintaining subdirectories with application names and version numbers, as shown below.

Extract downloaded contents:

For pre-compiled software, extract and move contents to your local software directory. For software that must be configured and compiled, extract and move contents to your source files directory.

Extraction flags:

  • tar.gz xvzf eXtract, Verbose, filter through gZip, using File
  • tar.bz2 xvjf …filter through bzip2 (j)

Extract pre-compiled software and move to local software directory:
[username@pegasus src]$ tar xvjf firefox-36.0.tar.bz2
[username@pegasus src]$ mv firefox-36.0 $HOME/local/firefox/36

The newly-extracted Firefox executable should now be located in ~/local/firefox/36/firefox
Pre-compiled binaries, skip to Updating PATH and creating symbolic links.

Extract source code and cd to new directory:
[username@pegasus src]$ tar xvzf autoconf-2.69.tar.gz
[username@pegasus src]$ cd autoconf-2.69
[username@pegasus autoconf-2.69]$ 

Source code, proceed to Configuring installation and compiling software.

Configuring installation and compiling software

We suggest using subdirectories with application names and version numbers, as shown below. There may be other configuration settings specific to your software.

Configure with local directory prefix (absolute path):

Configuration files may also be located in the bin (binary) directory, usually software/bin

[username@pegasus autoconf-2.69]$ ./configure --prefix=$HOME/local/autoconf/2.69
Make and install the software:
[username@pegasus autoconf-2.69]$ make
[username@pegasus autoconf-2.69]$ make install
...

If there are dependencies or conflicts, investigate the error output and try to resolve each error individually (install missing dependencies, check for specific flags suggested by software authors, check your local variables).

Updating PATH

PATH directories are searched in order. To ensure your compiled or downloaded software is found and used first, prepend the software executable location (usually in software/bin or software directories) to your PATH environment variable. Remember to add :$PATH to preserve existing environment variables.

Prepend software location to your PATH environment variable:
[username@pegasus ~]$ export PATH=$HOME/local/autoconf/2.69/bin:$PATH
Confirm by checking which software:
[username@pegasus ~]$ which autoconf
~/local/autoconf/2.69/bin/autoconf
Check software version:

Version flags may be software-dependent. Some common flags include --version, -v, and -V.

[username@pegasus ~]$ autoconf --version
autoconf (GNU Autoconf) 2.69
...

Creating symbolic links

To maintain multiple different versions of a program, use soft symbolic links to differentiate between the installation locations. Make sure the link and the directory names are distinct (example below). If local software has been kept in subdirectories with application names and version numbers, symlinks are not likely to conflict with other files or directories.

Create a distinctly-named symlink:

This symbolic link should point to the local software executable. The first argument is the local software executable location (~/local/firefox/36/firefox). The second argument is the symlink name and location (~/local/firefox36).

[username@pegasus ~]$ ln -s ~/local/firefox/36/firefox ~/local/firefox36
Append the local location to your PATH environment variable:

Remember to add :$PATH to preserve existing environment variables.

[username@pegasus ~]$ export PATH=$PATH:$HOME/local
Confirm both cluster copy and recently installed software:

The cluster copy of Firefox is firefox. The recently installed local copy is firefox36 from the symbolic links created above.

[username@pegasus ~]$ which firefox
/usr/bin/firefox
[username@pegasus ~]$ firefox --version
Mozilla Firefox 17.0.10

[username@pegasus ~]$ which firefox36
~/local/firefox36
[username@pegasus ~]$ firefox36 --version
Mozilla Firefox 36.0

Reminder – to launch Firefox, connect to Pegasus via SSH with X11 forwarding enabled.

Persistent PATH

To persist additions to your PATH variable, edit the appropriate profile configuration file in your home directory. For Bash on Pegasus, this is .bash_profile.

Update PATH in shell configuration (bash):

Use echo and the append redirect (>>) to update PATH in .bash_profile.

[username@pegasus ~]$ echo 'export PATH=$HOME/local/autoconf/2.69/bin:$PATH' >> ~/.bash_profile
[username@pegasus ~]$ echo 'export PATH=$PATH:$HOME/local' >> ~/.bash_profile

both in one command (note the newline special character \n directly in between the commands:

[username@pegasus ~]$ echo -e 'export PATH=$HOME/local/autoconf/2.69/bin:$PATH\nexport PATH=$PATH:$HOME/local' >> ~/.bash_profile

or edit the file directly:

[username@pegasus ~]$ vi ~/.bash_profile
...
PATH=$PATH:$HOME/bin
PATH=$HOME/local/autoconf/2.69/bin:$PATH
PATH=$PATH:$HOME/local
...
Reload shell configurations (Bash) and check PATH:

Look for the recently added path locations and their order.

[username@pegasus ~]$ source ~/.bash_profile
[username@pegasus ~]$ echo $PATH
/nethome/username/local/autoconf/2.69/bin:/share/opt/python/2.7.3/bin: ... :/share/sys65/root/sbin:/nethome/username/bin:/nethome/username/local

CCS does not install user software. Request cluster software installations or updates via the Software Request form.

Back to Top

Using Python Modules on Pegasus

Users are free to compile and install Python modules in their own home directories on Pegasus. Most Python modules can be installed with the --user flag using PIP, easy_install, or the setup.py file provided by the package. If you need a specific version of a Python module, we suggest using PIP with a direct link or downloading, extracting, and installing using setup.py. If you need to maintain multiple versions, see Python Virtual Environments (below).

The --user flag will install Python 2.7 modules here:   ~/.local/lib/python2.7/site-packages
Note the default location ~/.local is a hidden directory.

To specify a different location, use --prefix=$HOME/local/python2mods (or another path).
The above prefix flag example will install Python 2.7 modules here:   ~/local/python2mods/lib/python2.7/site-packages

Loading and Switching Python Modules

Confirm Python is loaded:

[username@pegasus ~]$ module list
Currently Loaded Modulefiles:
  1) perl/5.18.1             3) gcc/4.4.7(default)
  2) python/2.7.3(default)   4) share-rpms65

Switch Python modules:

[username@pegasus ~]$ module switch python/3.3.1
$ module list
Currently Loaded Modulefiles:
  1) perl/5.18.1          3) share-rpms65
  2) gcc/4.4.7(default)   4) python/3.3.1

Installing Python Modules with Package Managers

Install using PIP with --user:

[username@pegasus ~]$ pip install --user munkres
or install a specific version:
[username@pegasus ~]$ pip install --user munkres==1.0.7

Install using easy_install with --user:

[username@pegasus ~]$ easy_install --user munkres

Installing Downloaded Python Modules

Install using PIP with --user:

[username@pegasus ~]$ pip install --user https://pypi.python.org/packages/source/m/munkres/munkres-1.0.7.tar.gz
Downloading/unpacking https://pypi.python.org/packages/source/m/munkres/munkres-1.0.7.tar.gz
  Downloading munkres-1.0.7.tar.gz
  Running setup.py egg_info for package from https://pypi.python.org/packages/source/m/munkres/munkres-1.0.7.tar.gz
    
Cleaning up...

Install using setup.py with --user:

[username@pegasus ~]$ wget https://pypi.python.org/packages/source/m/munkres/munkres-1.0.7.tar.gz --no-check-certificate
[username@pegasus ~]$ tar xvzf munkres-1.0.7.tar.gz
[username@pegasus ~]$ cd munkres-1.0.7
[username@pegasus munkres-1.0.7]$ python setup.py --user install

Checking Module Versions

Launch Python and confirm module installation:

[username@pegasus ~]$ python
...
>>> import munkres
>>> print munkres.__version__
1.0.7
>>> CTRL-D (to exit Python)

Back to Top

Python Virtual Environments

Users can create their own Python virtual environments to maintain different module versions for different projects. Virtualenv is available on Pegasus for Python 2.7.3. By default, virtualenv does not include packages that are installed globally. To give a virtual environment access to the global site packages, use the --system-site-packages flag.

Creating Virtual Environments

These example directories do not need to be named exactly as shown.

Create a project folder, cd to the new folder (optional), and create a virtualenv:
[username@pegasus ~]$ mkdir ~/python2
[username@pegasus ~]$ cd ~/python2
[username@pegasus python2]$ virtualenv ~/python2/test1
PYTHONHOME is set.  You *must* activate the virtualenv before using it
New python executable in test1/bin/python
Installing setuptools, pip...done.
Create a virtualenv with access to global packages:
[username@pegasus python2]$ virtualenv --system-site-packages test2

Activating Virtual Environments

Activate the virtual environment with the source command and relative or absolute path/to/env/bin/activate. The environment name will precede the prompt.

[username@pegasus ~]$ source ~/python2/test1/bin/activate
(test1)[username@pegasus ~]$ which python
~/python2/test1/bin/python

Installing Python modules in Virtual Environments

Once the virtual environment is active, install Python modules normally with PIP, easy_install, or setup.py. Any package installed normally will be placed into that virtual environment folder and isolated from the global Python installation. Note that using --user or --prefix=... flags during module installation will place modules in those specified directories, NOT your currently active Python virtual environment.

(test1)[username@pegasus ~]$ pip install munkres

Deactivating Virtual Environments
(test1)[username@pegasus ~]$ deactivate
[username@pegasus ~]$ 

Comparing two Python Virtual Environments

PIP can be used to save a list of all packages and versions in the current environment (use freeze). Compare using sdiff to see which packages are different.

List the current environment, deactivate, then list the global Python environment:
(test1)[username@pegasus ~]$ pip freeze > test1.txt
(test1)[username@pegasus ~]$ deactivate
[username@pegasus ~]$ pip freeze > p2.txt
Compare the two outputs using sdiff:
[username@pegasus ~]$ sdiff p2.txt test1.txt
...
matplotlib==1.2.1                                             <
misopy==0.5.0                                                 | munkres==1.0.7
...
[username@pegasus ~]$ 

As seen above, the test1 environment has munkres installed (and no other global Python packages).

Recreating Python Virtual Environments

To recreate a Python virtual environment, use the r flag and the the saved list:

(test2)[username@pegasus ~]$ pip install -r test1.txt
Installing collected packages: munkres
  Running setup.py install for munkres
...
Successfully installed munkres
Cleaning up...
(test2)[username@pegasus ~]$ 

Python virtual environment wrapper

Users can install virtualenvwrapper in their own home directories to facilitate working with Python virtual environments. Once installed and configured, virtualenvwrapper can be used to create new virtual environments and to switch between your virtual environments (switching will deactivate the current environment). Virtualenvwrapper reads existing environments located in the WORKON_HOME directory.

Install a local copy of virtualenv with --user:

Recall that --user installs Python 2.7 modules in ~/.local/lib/python2.7/site-packages
To specify a different location, use --prefix=$HOME/local/python2mods (or another path).

[username@pegasus ~]$ pip install --user virtualenvwrapper
or
[username@pegasus ~]$ easy_install --user --always-copy virtualenvwrapper
Set virtual environment home directory and source:

WORKON_HOME should be the parent directory of your existing Python virtual environments (or another directory of your choosing). New Python virtual environments created with virtualenv will be stored according to this path. Set source to virtualenvwrapper.sh in the same location specified during installation.

[username@pegasus ~]$ export WORKON_HOME=$HOME/python2
[username@pegasus ~]$ source ~/.local/bin/virtualenvwrapper.sh
Create a virtual environment using virtualenvwrapper:

This will also activate the newly-created virtual environment.

[username@pegasus ~]$ mkvirtualenv test3
PYTHONHOME is set.  You *must* activate the virtualenv before using it
New python executable in test3/bin/python
Installing setuptools, pip...done.
(test3)[username@pegasus ~]$ 
Activate or switch to a virtual environment:
(test3)[username@pegasus ~]$ workon test1
(test1)[username@pegasus ~]$ 
Deactivate the virtual environment:
(test1)[username@pegasus ~]$ deactivate
[username@pegasus ~]$ 

Back to Top

Using Perl Modules on Pegasus

Users are free to compile and install Perl modules in their own home directories. Most Perl modules can be installed into a local library with CPAN and cpanminus. If you need a specific version, we suggest specifying the version or downloading, extracting, and installing using Makefile.PL.

Configuring a Local Library

Local libraries can be configured during initial CPAN configuration and by editing shell configuration files after installing the local::lib module.
By default, local::lib installs here: ~/perl5.

Configure with CPAN:

During initial CPAN configuration, answer yes to automatic configuration, local::lib to approach, and yes to append locations to your shell profile (Bash). Quit CPAN and source your shell configuration before running cpan again.

[username@pegasus ~]$ cpan
...
Would you like to configure as much as possible automatically? [yes] yes
...
Warning: You do not have write permission for Perl library directories.
...
What approach do you want?  (Choose 'local::lib', 'sudo' or 'manual')
 [local::lib] local::lib
...
local::lib is installed. You must now add the following environment variables
to your shell configuration files (or registry, if you are on Windows) and
then restart your command line shell and CPAN before installing modules:

PATH="/nethome/username/perl5/bin${PATH+:}${PATH}"; export PATH;
PERL5LIB="/nethome/username/perl5/lib/perl5${PERL5LIB+:}${PERL5LIB}"; export PERL5LIB;
PERL_LOCAL_LIB_ROOT="/nethome/username/perl5${PERL_LOCAL_LIB_ROOT+:}${PERL_LOCAL_LIB_ROOT}"; export PERL_LOCAL_LIB_ROOT;
PERL_MB_OPT="--install_base \"/nethome/username/perl5\""; export PERL_MB_OPT;
PERL_MM_OPT="INSTALL_BASE=/nethome/username/perl5"; export PERL_MM_OPT;

Would you like me to append that to /nethome/username/.bashrc now? [yes] yes
...
cpan[1]> quit
...
*** Remember to restart your shell before running cpan again ***
[username@pegasus ~]$ source ~/.bashrc

Configure after local::lib module installation:

If CPAN has already been configured, ensure local::lib is installed and the necessary environment variables have been added to your shell configuration files. Source your shell configuration before running cpan again.

[username@pegasus ~]$ cpan local::lib
Loading internal null logger. Install Log::Log4perl for logging messages
...
local::lib is up to date (2.000018).
[username@pegasus ~]$ echo 'eval "$(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)"' >> perl -~/.bashrc
[username@pegasus ~]$ source ~/.bashrc
...
Update CPAN:

Update CPAN, if necessary. This can be done from the Pegasus prompt or the CPAN prompt.

[username@pegasus ~]$ cpan CPAN
or
cpan[1]> install CPAN
...
Appending installation info to /nethome/username/perl5/lib/perl5/x86_64-linux-thread-multi/perllocal.pod
  ANDK/CPAN-2.10.tar.gz
  /usr/bin/make install  -- OK

cpan[2]> reload cpan
Confirm local library location:

Look for ~/perl5/... directories in $PATH and @INC.

[username@pegasus ~]$ echo $PATH
/share/opt/perl/5.18.1/bin:/nethome/username/perl5/bin:/share/lsf/9.1/linux2.6-glibc2.3-x86_64/etc:...
[username@pegasus ~]$ perl -e 'print "@INC"'
/nethome/username/perl5/lib/perl5/5.18.1/x86_64-linux-thread-multi /nethome/username/perl5/lib/perl5/5.18.1 ...

Installing Perl Modules

Once a local library has been installed and configured, CPAN modules will install to the local directory (default ~/perl5). The format for installing Perl modules with CPAN or cpanminus is Module::Name.

Install with CPAN:

Install from the Pegasus prompt or the CPAN prompt. Run cpan -h or perldoc cpan for more options.

[username@pegasus ~]$ cpan App::cpanminus
Loading internal null logger. Install Log::Log4perl for logging messages
Reading '/nethome/username/.cpan/Metadata'
...
or
[username@pegasus ~]$ cpan
cpan[1]> install App::cpanminus
Reading '/nethome/username/.cpan/Metadata'
...

To install a specific module version with cpan, provide the full distribution path.

[username@pegasus ~]$ cpan MIYAGAWA/App-cpanminus-1.7040.tar.gz

Install with cpanminus:

cpanminus is a CPAN module installation tool that will use your local library, if configured. Install from the Pegasus prompt with cpanm Module::Name. Run cpanm -h or perldoc cpanm for more options.

[username@pegasus ~]$ cpanm IO::All
--> Working on IO::All
Fetching http://www.cpan.org/authors/id/I/IN/INGY/IO-All-0.86.tar.gz ... OK
Configuring IO-All-0.86 ... OK
Building and testing IO-All-0.86 ... OK
Successfully installed IO-All-0.86
1 distribution installed

To install a specific module version with cpanm, provide either the full distribution path, the URL, or the path to a local tarball.

[username@pegasus ~]$ cpanm MIYAGAWA/App-cpanminus-1.7040.tar.gz
or
[username@pegasus ~]$ cpanm http://search.cpan.org/CPAN/authors/id/M/MI/MIYAGAWA/App-cpanminus-1.7040.tar.gz
or
[username@pegasus ~]$ cpanm ~/App-cpanminus-1.7040.tar.gz

Deactivating Local Library Environment Variables

To remove all directories added to search paths by local::lib in the current shell’s environment, use the --deactivate-all flag. Note that environment variables will be re-enabled in any sub-shells when using .bashrc to initialize local::lib.

[username@pegasus ~]$ eval $(perl -Mlocal::lib=--deactivate-all)
[username@pegasus ~]$ echo $PATH
/share/opt/perl/5.18.1/bin:...
[username@pegasus ~]$ perl -e 'print "@INC"'
/share/opt/perl/5.18.1/lib/site_perl/5.18.1/x86_64-linux-thread-multi ...

Source your shell configuration to re-enable the local library:

[username@pegasus ~]$ source ~/.bashrc
...
[username@pegasus ~]$ echo $PATH
/nethome/username/perl5/bin:/share/lsf/9.1/linux2.6-glibc2.3-x86_64/etc:...
[username@pegasus ~]$ perl -e 'print "@INC"'
/nethome/username/perl5/lib/perl5/5.18.1/x86_64-linux-thread-multi /nethome/username/perl5/lib/perl5/5.18.1 /nethome/username/perl5/lib/perl5/x86_64-linux-thread-multi /nethome/username/perl5/lib/perl5 ...

Back to Top

Running Matlab on Pegasus

Interactive Mode

There are several ways to run MATLAB commands/jobs interactively, with or without the graphical interface.

Graphical Interface Mode

To run MATLAB using graphical interface mode, connect with display forwarding. For more information about display forwarding, see Forwarding the Display (link).

Load and launch matlab on one of the interactive compute nodes as shown below. If you belong to more than one project, specify the projectID as well.

[username@pegasus ~]$ module load matlab
[username@pegasus ~]$ bsub -Is -q interactive -XF -P projectID matlab

Once the interactive MATLAB graphical desktop is loaded, you can then run MATLAB commands or scripts in the MATLAB command window. The results will be shown in the MATLAB command window and the figure/plot will be displayed in new graphical windows on your computer. See examples below.

>> x = rand(1,100);
>> plot(x);
>>
>> x = [0: pi/10: pi];
>> y = sin(x);
>> z = cos(x);
>> figure;
>> plot(x, y);
>> hold('on');
>> plot(x, z, '--');

Graphical Interactive Mode with no graphical desktop window

Running MATLAB in a full graphical mode may get slow depending on the network load. Running it with -nodesktop option will use your current terminal window (in Linux/Unix) as a desktop, while allowing you still to use graphics for figures and editor.

[username@pegasus ~]$ module load matlab
[username@pegasus ~]$ bsub -Is -q interactive -XF -P projectID matlab -nodesktop

Non-Graphical interactive Mode

If your MATLAB commands/jobs do not need to show graphics such as figures and plots, or to use a built-in script editor, run the MATLAB in the non-graphical interactive mode with -nodisplay.

Open a regular ssh connection to Pegasus.

[username@pegasus ~]$ module load matlab
[username@pegasus ~]$ bsub -Is -q interactive -P projectID matlab -nodisplay

This will bring up the MALAB command window:

                            < M A T L A B (R) >
                  Copyright 1984-2013 The MathWorks, Inc.
                    R2013a (8.1.0.604) 64-bit (glnxa64)
                             February 15, 2013


No window system found.  Java option 'Desktop' ignored.

To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
>> 

To exit, type exit or quit. Again, remember to import the prepared LSF configuration file mentioned above if you want to use MATLAB parallel computing.

Batch Processing

For off-line non-interactive computations, submit the MATLAB script to the LSF scheduler using the bsub command. For more information about job scheduling, see Scheduling Jobs (link). Example single-processor job submission:

[username@pegasus ~]@ bsub < example.job
example.job

#BSUB -J example #BSUB -q general #BSUB -P projectID #BSUB -n 1 #BSUB -o example.o%J #BSUB -e example.e%J matlab -nodisplay -r my_script

In this example, “my_script” corresponds to “my_script.m” in the current working directory.

After the job is finished, the results will be saved in the output file named “example.o######” where “######” is a jobID number assigned by LSF when you submit your job.

Parallel Computations

MATLAB has software products to enable parallel computations for multi-core computers as well as for multiple-node computer clusters. The latter case scenario requires a job scheduler, such as LSF on Pegasus cluster.

The MATLAB product for the parallel processing that uses the cores of the same node is the “Distributed Computing Toolbox/DCT” (also appears in MATLAB documentation under the name of “Parallel Computing Toolbox”). Licensed MATLAB software product for a computer cluster is called “Distributed Computing Engine/DCE” (also appears in documentation as “MATLAB Distributed Computing Server”).

Single-node parallel MATLAB jobs (up to 16 cpus)

For a single-node parallel job, MATLAB Distributed Computing Toolbox (licensed software) is used. It has a build-in default MATLAB cluster profile ‘local’, from which the pool of MatlabWorkers can be reserved for computations. The default number of MatlabWorkers is 12. You can specify up to 15 on a single Pegasus node using the general queue, and 16 cpus using the parallel queue. For more information about queue and parallel resource distribution requirements, see Scheduling Jobs (link).

Refer to MATLAB documentation on the ways to adapt your script for multi-processor calculations. One of the parallel tools in MATLAB is the parfor loop replacing the regular for loop, and in the example is given below:

%==============================================================
% dct_example.m
% Distributed Computing Toolbox (DCT)
% Example: Print datestamp within a parallel "parfor" loop
%==============================================================
%% Create a parallel pool of MatlabWorkers on the current working node:
parpool('local',16);
% The test loop size
N = 40;       
tstart = tic();
parfor(ix=1:N)
  ixstamp = sprintf('Iteration %d at %s\n', ix, datestr(now));
  disp(ixstamp);
  pause(1);
end
cputime=toc(tstart);
toctime= sprintf('Time used is %d seconds', cputime);
disp(toctime)
%% delete current parallel pool:
delete(gcp)  

Multi-node parallel MATLAB jobs (16-32 cpus)

For running multi-processor MATLAB jobs that involve 16+ cpus and more than a single node, MATLAB Distributed Computer Engine (licensed software) is used, with currently 32 licenses available on Pegasus. These jobs must be submitted to the parallel queue with the appropriate ptile resource distribution. For more information about queue and resource distribution requirements, see Scheduling Jobs (link).

The parallel LSF MATLAB cluster also needs to be configured. After loading the matlab module, import the default LSF parallel configuration as following:

[username@pegasus ~]$ matlab -nodisplay -r "parallel.importProfile('/share/opt/MATLAB/etc/LSF1.settings');exit";reset

This command only needs to be run once. It imports the cluster profile named ‘LSF1’ that is configured to use up to 32 MatlabWorkers and to submit MATLAB jobs to the parallel pegasus queue. This profile does not have a projetID associated with the job, and you may need to coordinate the project name for the LSF job submission. This can be done by running the following script (only once!) during your matlab session:

%% conf_lsf1_project_id.m
%% Verify that LSF1 profile exists, and indicate the current default profile:
[allProfiles,defaultProfile] = parallel.clusterProfiles()
%% Define the current cluster object using LSF1 profile
myCluster=parcluster('LSF1')
%% View current submit arguments:
get(myCluster,'SubmitArguments')
%% Set new submit arguments, change projectID below to your current valid project:
set(myCluster,'SubmitArguments','-q general -P projectID')
%% Save the cluster profile:
saveProfile(myCluster)
%% Set the 'LSF1' to be used as a default cluster profile instead of a 'local'
parallel.defaultClusterProfile('LSF1');
%% Verify the current profiles and the default:
[allProfiles,defaultProfile] = parallel.clusterProfiles()

The above script also reviews your current settings of the cluster profiles. You can now use the cluster profile for distributed calculations on up to 32 cpus, for example, to create a pool of MatlabWorkers for a parfor loop:

%=========================================================
% dce_example.m
% Distributed Computing Engine (DCE) 
% Example: Print datestamp within a parallel "parfor" loop
%=========================================================
myCluster=parcluster('LSF1')
% Maximum number of MatlabWorkers is 32 (number of MATLAB DCE Licenses)
parpool(myCluster,32);
% The test loop size
N = 40;  
tstart = tic();
parfor(ix=1:N)
  ixstamp = sprintf('Iteration %d at %s\n', ix, datestr(now));
  disp(ixstamp);
  pause(1);
end
cputime=toc(tstart);
toctime= sprintf('Time used is %d seconds', cputime);
disp(toctime)
delete(gcp)

Please see MATLAB documentation on more ways to parallelize your code.

There may be other people running Distributed Computing Engine and thus using several licenses. Please check the license count as following (all in a single line):

[username@pegasus ~]$ /share/opt/MATLAB/R2013a/etc/lmstat -S MLM -c /share/opt/MATLAB/R2013a/licenses/network.lic

Find the information about numbers of licenses used for the “Users of MATLAB_Distrib_Comp_Engine”, “Users of MATLAB”, and “Users of Distrib_Computing_Toolbox”.

Note on Matlab cluster configurations

After importing the new cluster profile, it will remain in your available cluster profiles. Validate using the parallel.clusterProfiles() function. You can create, change, and save profiles using SaveProfile and SaveAsProfile methods on a cluster object. In the examples, “myCluster” is the cluster object. You can also create, import, export, delete, and modify the profiles through the “Cluster Profile Manager” accessible via MATLAB menu in a graphical interface. It is accessed from the “HOME” tab in the GUI desktop window under “ENVIRONMENT” section: ->”Parallel”->”Manage Cluster Profiles”

Cluster Profile Manager

You can also create your own LSF configuration from the Cluster Profile Manager. Choose “Add”->”Custom”->”3RD PARTY CLUSTER PROFILE”->”LSF” as shown below:

Cluster Profile Manager: new LSF cluster

… and configure to your needs:

New LSF cluster in Matlab

Running R on Pegasus

To run the R programming language on Pegasus, first load an R module with module load R/version. Use the command module avail to see a list of available software, including R versions. For more information about Pegasus software, see Software on the Pegasus Cluster (link).

Batch R

To run a batch R file on Pegasus compute nodes, submit the file to LSF with R CMD BATCH filename.R. Remember to include the -P flag and your project, if you have one.

[username@pegasus ~]$ module load R/2.15.2
[username@pegasus ~]$ bsub -q general -P projectID R CMD BATCH example1.R
Job is submitted to <projectID> project.
Job <6101046> is submitted to queue <general>.

See example1.R below. For more information about bsub and scheduling jobs, see Scheduling Jobs on the Pegasus Cluster (link).

Batch jobs can also be submitted to LSF with script files.

example.job

#!/bin/bash #BSUB -J R_job # name of your job #BSUB -R "span[hosts=1]" # request run script on one node #BSUB -q general # request run on general queue #BSUB -n 1 # request 1 core #BSUB -W 2 # request 2 minutes of runtime #BSUB -P projectID # your projectID R CMD BATCH example1.R # R command and your batch R file

[username@pegasus ~]$ bsub < example.job

Interactive R

Load an R module, then submit an interactive R session with bsub -Is. Specify your project with the -P flag.

[username@pegasus ~]$ module load R/2.15.2
[username@pegasus ~]$ bsub -Is -q interactive R
or
[username@pegasus ~]$ bsub -Is -P hpc R

R packages

From the R prompt, install any R package to your personal R library with the standard install.package() R command. Choose y when asked about using a personal library, and y again if asked to create one.

>  install.packages("doParallel", repos="http://R-Forge.R-project.org")
Warning in install.packages("doParallel", repos = "http://R-Forge.R-project.org") :
  'lib = "/share/opt/R/3.1.2/lib64/R/library"' is not writable
Would you like to use a personal library instead?  (y/n) y
Would you like to create a personal library
~/R/x86_64-unknown-linux-gnu-library/3.1
to install packages into?  (y/n) y
...

R file examples
# create graphical output file
pdf("example1.pdf")

# Define two vectors v1 and v2
v1 <- c(1, 4, 7, 8, 10, 12)
v2 <- c(2, 8, 9, 10, 11, 15)

# Creat some graphs
hist(v1)
hist(v2)
pie(v1)
barplot(v2)

# close the file
dev.off() 

If you have forwarded your display, open the pdf from the Pegasus prompt with evince.

[username@pegasus ~]$ evince example1.pdf

example1.pdf

Back to Top

Running SAS on Pegasus

SAS can be run on Pegasus in Non-interactive/Batch and Interactive/Graphical modes.

Non-Interactive Batch Mode

In batch mode, SAS jobs should be submitted via LSF using the bsub command. A sample LSF script file named scriptfile to submit SAS jobs on Pegasus may include the following lines:

scriptfile

#BSUB -J jobname #BSUB -o jobname.o%J #BSUB -e jobname.e%J sas test.sas

where “test.sas” is an SAS program file.

Type the following command to submit the job:

[username@pegasus ~]$ bsub < scriptfile

For general information about how to submit jobs via LSF, see Scheduling Jobs on Pegasus (direct link).

Interactive Graphical Mode

To run SAS interactively, first forward the display (direct link). Load the SAS module and use the interactive queue to launch the application.

[username@pegasus ~]$ module load sas
[username@pegasus ~]$ module load java

Submit job to the interactive queue:

[username@pegasus ~]$ bsub -q interactive -P myproject -Is -XF sas
Job is submitted to project.
Job is submitted to queue .

Notice the -P flag in the above bsub command. If you do not specify your project, you will receive an error like the one below:

[username@pegasus ~]$ bsub -q interactive -Is -XF sas
Error: Your account has multiple projects: project1 project2.
Please specify a project by -P option and resubmit
Request aborted by esub. Job not submitted.

Back to Top

SVN - Subversion

Subversion Server

User access to Subversion server is managed through CCS LDAP accounts and local ACLs. Authorized users can access SVN repositories with their CCS accounts.

command line access: http://web.ccs.miami.edu/repos/REPOSITORYNAME
web access: http://web.ccs.miami.edu/websvn

Subversion Client

For Windows, we recommend TortoiseSVN. This Subversion client seamlessly integrates into Windows Explorer. A right-click provides with the most common Subversion commands.

For Mac, download the latest Subversion client from collab.net. Extract the SVN binary to /usr/local/bin. Consider SvnX, a good open source front-end for Subversion. As with all Mac apps, download the dmg file, double click the file if it does not auto mount, then drag the SvnX application to your system’s Application directory. See this tutorial for help configuring SvnX: Getting Started with SvnX.

Most Linux distributions already have the SVN client. If not, run sudo yum install subversion – more information here: CentOS Subversion HowTo. Note, this may take a good ten minutes.

Basic Subversion commands

Include the -m flag and a message with SVN commits, adds, and imports. For more information about Subversion commands, run svn help at the command-line prompt.

Command Description
svn list repo_address List files in a repository.
svn import /path/to/directory repo_address -m 'tree description' Add and commit all content under directory to the specified repo, with comments (-m flag). Run svn checkout after import to create working copies on your machine.
svn checkout repo_address
svn co repo_address
Checkout a repository by creating a working copy (snapshot) on your machine.
A repository must be checked out to run the below commands.
svn add filename_or_directory -m 'description' Add a new file or the contents of a new directory to the current working copy, with comments (-m flag).
Commit or check-in after adding files to update the repo.
svn delete filename_or_directory Delete file or directory from the current working copy.
Commit or check-in after deleting files to update the repo.
svn status optional filenames or directories Review all modified files, or specify multiple file or directory names. Add the --verbose flag to see details.
svn commit filename_or_directory -m 'explanation'
svn ci filename_or_directory -m 'explanation of changes'
Commit or check-in changes to a specific file or all files in directory, with comments (-m flag)
Review local changes with svn status before committing them.
svn commit -m 'explanation'
svn ci -m 'explanation of changes'
Commit or check-in all changes, with comments (-m flag)
Review local changes with svn status before committing them.
svn revert optional filenames or directories
svn revert -R .
Revert any un-committed changes to the most recent snapshot versions.
Revert all un-committed changes with -R .
This does not delete files that are not managed by SVN (status ?)
svn update optional filenames or directories
svn up optional filenames or directories
Update your working copies with versions from the repository, or specify multiple file names.
SVN will attempt to merge any changes.
svn diff optional filenames or directories
svn diff revision1:revision2 optional filenames or directories
Review differences between your current working copy and the snapshot, or specify revision numbers. Optionally specify multiple file names.

Basic SVN Usage

Once your repository is available, use svn import to populate it with content from a directory on your local machine. Remember to check out the repository after, to create SVN-managed working copies on your machine.

[username@pegasus test]$ svn import test http://web.ccs.miami.edu/repos/mydept/myrepo -m 'adding all content under test'
Adding         test/file1.test
Adding         test/file2.test
Adding         test/file3.test
Committed revision 41.

To create a working copy (private snapshot) of all files in a repository on your local computer (in a directory with the repository name) use svn checkout. This initial copy is your snapshot. Subversion will keep track of changes in your working copy, which are pending with respect to the repository until committed with svn commit (svn ci). It is good practice to review your local changes with svn status before committing them.

[username@pegasus ~]$ svn checkout http://web.ccs.miami.edu/repos/mydept/myrepo
A    myrepo/test/file1.test
A    myrepo/test/file2.test
A    myrepo/test/file3.test
Checked out revision 42.
[username@pegasus ~]$ cd myrepo/test
[username@pegasus test]$ ls
file1.test  file2.test  file3.test

After editing your working copy of repository files, commit (upload) all or some changes to the Subversion server with svn commit (or svn ci). Take the time to write a decent comment explaining your changes.

[username@pegasus test]$ svn ci file3.test -m 'updated equation in line 19'
Sending        file3.test
Transmitting file data .
Committed revision 43.

Add your own files or directories to your local working copy with svn add. Run svn ci after adding, to commit the changes to the repository. Take the time to write a decent comment explaining your changes.

[username@pegasus test]$ svn add file4.test
A         file4.test
[username@pegasus test]$ svn commit file4.test -m 'adding a new file'
Adding         file4.test
Transmitting file data ..
Committed revision 44.

[usrname@pegasus test]$ svn add testtree
A         testtree
A         testtree/subfile1.test
A         testtree/subfile2.test
[username@pegasus test]$ svn ci testtree -m 'committing additional testtree directory'
Adding         testtree
Adding         testtree/subfile1.test
Adding         testtree/subfile2.test
Transmitting file data ..
Committed revision 45.

Delete files or directories from your local working copy with svn delete. Run svn ci after deleting, to commit the changes to the repository. Take the time to write a decent comment explaining your changes.

[username@pegasus test]$ svn delete testtree/subfile2.test
D         testtree/subfile2.test
[username@pegasus test]$ svn ci -m 'committing deletion of subfile2.test'
Deleting       testtree/subfile2.test

Committed revision 46.

[username@pegasus test]$ svn delete testtree
D         testtree/subfile1.test
D         testtree
[username@pegasus test]$ svn ci -m 'committing deletion of directory testtree and contents'
Deleting       testtree

Committed revision 47.

Review modifications made to your local working copy with svn status. Use the --verbose flag to show details, including revision and owner information. Specify files or directories with optional arguments.

[username@pegasus test]$ svn status
?       file4.test
A       file3.test
M       file1.test

In this example, file1.test has been modified (M), file3.test has been added to the working copy (not the repo), and file4.test has not been added to the working copy (?). file2.test matches the repository version (all files are shown with --verbose flag and no arguments):

[username@pegasus test]$ svn status --verbose
               47      47 username      .
?                                        file4.test
A                                        file3.test
               47      47 username      file2.test
M              47      47 username      file1.test

Show verbose status for only file4.test and file1.test:

[username@pegasus test]$ svn status file1.test file4.test --verbose
M              47      47 username      file1.test
?                                       file34.test

Add file4.test to the local working copy, then commit the updates and additions separately:

[username@pegasus test]$ svn add file4.test
A         file4.test
[username@pegasus test]$ svn status
A       file4.test
A       file3.test
M       file1.test
[username@pegasus test]$ svn ci file1.test -m 'updating file1.test'
Sending        file1.test
Transmitting file data .
Committed revision 48.
[username@pegasus test]$ svn ci -m 'adding 2 test files'
Adding         file3.test
Adding         file4.test
Transmitting file data ..
Committed revision 49.

Revert any un-committed changes to your local working copy with svn revert. This will return the specified files or directories in the working copy to the checked-out snapshot. Revert all with -R . (this will not delete any new files with ? status).

[username@pegasus test]$ svn status
?       file4.test
M       file2.test
M       file3.test
M       file1.test
[username@pegasus test]$ svn revert file1.test
Reverted 'file1.test'
[username@pegasus test]$ svn status
?       file4.test
M       file2.test
M       file3.test
[username@pegasus test]$ svn revert -R .
Reverted 'file2.test'
Reverted 'file3.test'
[username@pegasus test]$ svn status
?       file4.test

Update your local working copy to the current repository version with svn update (svn up). SVN will attempt to merge any changes on the server with committed changes to your local working copy. Specify files with optional arguments.

[username@pegasus test]$ svn up file2.test
U    file2.test
Updated to revision 50.
[username@pegasus test]$ svn up
U    file1.test
U    file3.test
U    file4.test
Updated to revision 50.

Review differences between two versions with svn diff. Without arguments, this shows the differences between your local working copies and the snapshot (your most recent retrieval from the repository). Specify revisions with -r rev1:rev2 and files or directories with optional arguments. Revision order matters for svn diff -r output.

In this example, file2.test starts empty. A line has been added to the local working copy. The differences betwen the local working copy and the snapshot are shown:

[username@pegasus test]$ svn diff file2.test
Index: file2.test
===================================================================
--- file2.test	(revision 50)
+++ file2.test	(working copy)
@@ -0,0 +1 @@
+username added this line to local working copy

The local file2.test is then committed to the repository, and differences between revisions are shown. Note that the order of revisions affects output format (not output content).

[username@pegasus test]$ svn ci file2.test -m 'updating repo'
Sending        file2.test
Transmitting file data .
Committed revision 51.  

[username@pegasus test]$ svn diff -r 50:51 file2.test
Index: file2.test
===================================================================
--- file2.test	(revision 50)
+++ file2.test	(revision 51)
@@ -0,0 +1 @@
+username added this line to local working copy

[username@pegasus test]$ svn diff -r 51:50 file2.test
Index: file2.test
===================================================================
--- file2.test	(revision 51)
+++ file2.test	(revision 50)
@@ -1 +0,0 @@
-username added this line to local working copy

Back to Top

Allinea - debugging and profiling

Profile and Debug with Allinea Forge, the new name for the unified Allinea MAP and Allinea DDT tools. See the user guide PDFs below for Allinea Forge and Performance Reports, available as modules on Pegasus.

Allinea 6.1-Forge guide [PDF download]

Allinea 6.1-PR guide [PDF download]

Scheduling Jobs on Pegasus

Pegasus currently uses the LSF resource manager to schedule all compute resources. LSF (load sharing facility) supports over 1500 users and over 200,000 simultaneous job submissions. Jobs are submitted to queues, the software categories we define in the scheduler to organize work more efficiently. LSF distributes jobs submitted by users to our over 340 compute nodes according to queue, user priority, and available resources. You can monitor your job status, queue position, and progress using LSF commands shown in the table below.

You may compile and test jobs on login nodes. However, any jobs exceeding 30 minutes of run time or using excessive resources on the login nodes will be terminated and the CCS account responsible for those jobs may be suspended (Policies).

Reserve an appropriate amount of resources through LSF for your jobs. Jobs with insufficient resource allocations interfere with cluster performance and the CCS account responsible for those jobs may be suspended (Policies).
If you do not know the resources your jobs need, use the debug queue to benchmark your jobs. More on Pegasus Queues (link) and LSF Job Scripts (link) below.

Submitting Batch Jobs to LSF – Overview

Batch jobs are self-contained programs that require no intervention to run. Batch jobs are defined by resource requirements such as how many cores, how much memory, and how much time they need to complete. These requirements can be submitted via command line flags or a script file. Detailed information about LSF commands and example script files can be found later in this guide.

  1. Create a job scriptfile

    Include a job name -J, the information LSF needs to allocate resources to your job, and names for your output and error files.

    scriptfile
    
    #BSUB -J test #BSUB -q general #BSUB -P myproject #BSUB -o %J.out ...

  2. Submit your job to the appropriate project and queue with bsub < scriptfile

    Upon submission, a jobID and the queue name are returned.

    [username@pegasus ~]$ bsub < scriptfile 
    Job <6021006> is submitted to queue <general>.

  3. Monitor your jobs with bjobs

    Flags can be used to specify a single job or another user’s jobs.

    [username@pegasus ~]$ bjobs
    JOBID  USER   STAT  QUEUE    FROM_HOST  EXEC_HOST   JOB_NAME  SUBMIT_TIME
    4225   usernam   RUN   general  m1       16*n060     testjob   Mar  2 11:53

  4. Examine job output files

    Once your job has completed, view output information.

    [username@pegasus ~]$ cat test.out
    Sender: LSF System <lsfadmin@n069.pegasus.edu>
    Subject: Job 6021006: <test> in cluster <mk2> Done
    Job <test> was submitted from host <login4.pegasus.edu> by user <username> in cluster <mk2>.
    Job was executed on host(s) <8*n069>, in queue <general>, as user <username> in cluster <mk2>.
    ...

Back to Top

Pegasus Job Queues

Pegasus queues are organized using limits like job size, job length, job purpose, and project. In general, users run jobs on Pegasus with equal resource shares. Current or recent resource usage lowers the priority applied when LSF assigns resources for new jobs from a user’s account.

Parallel jobs are more difficult to schedule as they are inherently larger. Serial jobs can “fit into” the gaps left by larger jobs if serial jobs use short enough run time limits and small enough numbers of processors.

Queue Name Processors
(Cores)
Memory Wall time
default / max
Description
general 15- 25GB max 1 day / 7 days jobs up to 15 cores, up to 25GB memory
parallel 16+ 25GB max 1 day / 7 days parallel jobs requiring 16 or more cores, up to 25GB memory
requires resource distribution -R "span[ptile=16]"
bigmem 64 max 250GB max 4 hours / 5 days jobs requiring nodes with expanded memory
debug 64 max 25GB Max 30 mins / 30 mins job debugging
interactive 15- 250GB max 6 hours / 1 day interactive jobs only
max 1 job per user
gpu 320 max 1 day / 7 days gpu debugging
restricted queue
phi 320 max 1 day / 7 days phi debugging
restricted queue

Back to Top

LSF Commands

LSF 9.1.1 Documentation

Common LSF commands and descriptions:

Command Purpose
bsub Submits a job to LSF.
Define resource requirements with flags.
bsub < scriptfile Submits a job to LSF via script file.
The redirection symbol < is required when submitting a job script file
bjobs Displays your running and pending jobs.
bhist Displays historical information about your finished jobs.
bkill Removes/cancels a job or jobs from the class.
bqueues Shows the current configuration of queues.
bhosts Shows the load on each node.
bpeek Displays stderr and stdout from your unfinished job.

Scheduling Jobs

The command bsub will submit a job for processing. You must include the information LSF needs to allocate the resources your job requires, handle standard I/O streams, and run the job. For more information about flags, type bsub -h at the Pegasus prompt. Detailed information can be displayed with man bsub. On submission, LSF will return the job id which can be used to keep track of your job.

[username@pegasus ~]$ bsub -J jobname -o %J.out -e %J.err -q general -P myproject myprogram
Job <2607> is submitted to general queue .

The Job Scripts section has more information about organizing multiple flags into a job script file for submission.

Monitoring Jobs
bjobs

The commands bjobs displays information about your own pending, running, and suspended jobs.

[username@pegasus ~]$ bjobs
JOBID  USER   STAT  QUEUE    FROM_HOST  EXEC_HOST   JOB_NAME  SUBMIT_TIME
4225   usernam   RUN   general  m1       16*n060     testjob   Mar  2 11:53
                                         16*n061
                                         16*n063
                                         16*n064

For details about your particular job, issue the command bjobs -l jobID where jobID is obtained from the JOBID field of the above bjobs output. To display a specific user’s jobs, use bjobs -u username. To display all user jobs in paging format, pipe output to less:

[username@pegasus ~]$ bjobs -u all | less
JOBID     USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
5990529   axt651  RUN   interactiv login4.pega n002        bash       Feb 13 15:23
6010636   zxh69   RUN   general    login4.pega 16*n178     *acsjob-01 Feb 23 11:36
                                               16*n180
                                               16*n203
                                               16*n174
6014246   swishne RUN   interactiv n002.pegasu n002        bash       Feb 24 14:10
6017561   asingh  PEND  interactiv login4.pega             matlab     Feb 25 14:49
...

bhist

bhist displays information about your recently finished jobs. CPU time is not normalized in bhist output. To see your finished and unfinished jobs, use bhist -a.

bkill

bkill kills the last job submitted by the user running the command, by default. The command bkill jobID will remove a specific job from the queue and terminate the job if it is running. bkill 0 will kill all jobs belonging to current user.

[username@pegasus ~]$ bkill 4225
Job <4225> is being terminated

On Pegasus (Unix), SIGINT and SIGTERM are sent to give the job a chance to clean up before termination, then SIGKILL is sent to kill the job.

bqueues

bqueues displays information about queues such as queue name, queue priority, queue status, job slot statistics, and job state statistics. CPU time is normalized by CPU factor.

[username@pegasus ~]$ bqueues
QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP 
bigmem          500  Open:Active       -   16    -    -  1152  1120    32     0
visx            500  Open:Active       -    -    -    -     0     0     0     0
hihg            500  Open:Active       -    -    -    -     0     0     0     0
hpc             300  Open:Active       -    -    -    -  2561  1415  1024     0
debug           200  Open:Active       -    -    -    -     0     0     0     0
gpu             200  Open:Active       -    -    -    -     0     0     0     0
...
general         100  Open:Active       -    -    -    -  9677  5969  3437     0
interactive      30  Open:Active       -    4    -    -    13     1    12     0

bhosts

bhosts displays information about all hosts such as host name, host status, job state statistics, and jobs lot limits. bhosts -s displays information about numeric resources (shared or host-based) and their associated hosts. bhosts hostname displays information about an individual host and bhosts -w displays more detailed host status. closed_Full means the configured maximum number of running jobs has been reached (running jobs will not be affected), no new job will be assigned to this host.

[username@pegasus ~]$ bhosts -w | less
HOST_NAME          STATUS       JL/U    MAX  NJOBS    RUN  SSUSP  USUSP    RSV 
n001               ok              -     16     14     14      0      0      0
n002               ok              -     16      4      4      0      0      0
...
n342               closed_Full     -     16     16     12      0      0      4
n343               closed_Full     -     16     16     16      0      0      0
n344               closed_Full     -     16     16     16      0      0      0

bpeek

Use bpeek jobID to monitor the progress of a job and identify errors. If errors are observed, valuable user time and system resources can be saved by terminating an erroneous job with bkill jobID. By default, bpeek displays the standard output and standard error produced by one of your unfinished jobs, up to the time the command is invoked. bpeek -q queuename operates on your most recently submitted job in that queue and bpeek -m hostname operates on your most recently submitted job dispatched to the specified host. bpeek -f jobID display live outputs from a running job and it can be terminated by Ctrl-C (Windows & most Linux) or Command-C (Mac).

Examining Job Output

Once your job has completed, examine the contents of your job’s output files. Note the script submission under User input, whether the job completed, and the Resource usage summary.

[username@pegasus ~]$ cat test.out
Sender: LSF System <lsfadmin@n069.pegasus.edu>
Subject: Job 6021006: <test> in cluster <mk2> Done
Job <test> was submitted from host <login4.pegasus.edu> by user <username> in cluster <mk2>.
Job was executed on host(s) <8*n069>, in queue <general>, as user <username> in cluster <mk2>.
...
Your job looked like:
------------------------------------------------------------
# LSBATCH: User input
#!/bin/sh
#BSUB -n 16
#BSUB -J test
#BSUB -o test.out
...
------------------------------------------------------------
Successfully completed.
Resource usage summary:
CPU time : 2.26 sec.
Max Memory : 30 MB
Average Memory : 30.00 MB
...
PS:
Read file <test.err> for stderr output of this job.

Back to Top

LSF Job Scripts

The command bsub < ScriptFile will submit the given script for processing. Your script must contain the information LSF needs to allocate the resources your job requires, handle standard I/O streams, and run the job. For more information about flags, type bsub -h or man bsub at the Pegasus prompt. Example scripts and descriptions are below.

You must be a member of a project to submit jobs to it. If you are not a member of the specified project, your job will be submitted to the ‘default’ project which is assigned limited cluster resources. See Projects (link) for more information.

On submission, LSF will return the jobID which can be used to track your job.

[username@pegasus ~]$ bsub < test.job
Job <4225> is submitted to the default queue <general>.

Example script for a serial Job

test.job

#!/bin/bash #BSUB -J myserialjob #BSUB -P myproject #BSUB -o %J.out #BSUB -e %J.err #BSUB -W 1:00 #BSUB -q general #BSUB -n 1 #BSUB -R "rusage[mem=128]" #BSUB -B #BSUB -N #BSUB -u myemail@miami.edu # # Run serial executable on 1 cpu of one node cd /path/to/scratch/directory ./test.x a b c

Here is a detailed line-by-line breakdown of the keywords and their assigned values listed in this script:

ScriptFile_keywords

#!/bin/bash specifies the shell to be used when executing the command portion of the script. The default is Bash shell. BSUB -J serialjob assign a name to job. The name of the job will show in the bjobs output. #BSUB -P myproject specify the project to use when submitting the job. This is required when a user has more than one project on Pegasus. #BSUB -e %J.err redirect std error to a specified file #BSUB -W 1:00 set wall clock run time limit of 1 hour, otherwise queue specific default run time limit will be applied. #BSUB -q general specify queue to be used. Without this option, default 'general' queue will be applied. #BSUB -n 1 specify number of processors. In this job, a single processor is requested. #BSUB -R "rusage[mem=128]" specify that this job requests 128 megabytes of RAM. Without this, a default RAM setting will be applied. #BSUB -B send mail to specified email when the job is dispatched and begins execution. #BSUB -u example@miami.edu send notification through email to example@miami.edu. #BSUB -N send job statistics report through email when job finishes.

Example scripts for parallel jobs

We recommend using Intel MPI unless you have specific reason for using OpenMP.  Intel MPI scales better and has better performance than OpenMP.

Submit parallel jobs to the parallel job queue with -q parallel.

For optimum performance, the default resource allocation on the parallel queue is ptile=16. This requires the LSF job scheduler to allocate 16 processors per host, ensuring all processors on a single host are used by that job. Without prior authorization, any jobs using a number other than 16 will be rejected from the parallel queue. Parallel job performance may be affected, or even interrupted, by other badly-configured jobs running on the same host.

Example script for Intel/Intel MPI
testparai.job

#!/bin/bash #BSUB -J mpijob #BSUB -o %J.out #BSUB -e %J.err #BSUB -W 1:30 #BSUB -q parallel #BSUB -n 32 #BSUB -R "span[ptile=16]" # mpiexec foo.exe

foo.exe is the mpi executable name. It can be followed by its own argument list.

Example script for MPI/OpenMP
testparao.job

#!/bin/bash #BSUB -J mpijob #BSUB -o %J.out #BSUB -e %J.err #BSUB -W 1:30 #BSUB -q parallel #BSUB -n 32 #BSUB -R "span[ptile=16]" # mpiexec --mca btl self,sm,openib foo.exe

The command line is similar to Intel MPI job above. Option --mca self,sm,openib tells OpenMP to use lookback, shared memory, and openib for inter-process communication.

Back to Top

LSF Interactive Jobs

HPC clusters primarily take batch jobs and run them in the background—users do not need to interact with the job during the execution. However, sometimes users do need to interact with the application. For example, the application needs the input from the command line or waits for a mouse event in X windows. Use bsub -Is -q interactive to launch interactive work on Pegasus.

[username@pegasus ~]$ bsub -Is -q interactive matlab -nodisplay

or

[username@pegasus ~]$ bsub -Is -q interactive -XF $(java -jar ~/.local/apps/ImageJ/ij.jar -batch ~/.local/apps/ImageJ/macros/screenmill.txt)

Upon exiting the interactive job, you will be returned to one of the login nodes.

Interactive Job Utilizing X11 client

Additionally, the interactive queue can run X11 jobs. The bsub -XF option is used for X11 jobs, for example:

[username@pegasus ~]$ bsub -Is -q interactive -XF matlab
Job <50274> is submitted to queue <interactive>.
<<ssh X11 forwarding job>>
<<Waiting for dispatch ...>>
<<Starting on n003.pegasus.edu>> 

Upon exiting the X11 interactive job, you will be returned to one of the login nodes.

To run an X11 application, establish an X tunnel with SSH when connecting to Pegasus. For example,

ssh -X username@pegasus.ccs.miami.edu

Note that by default, the auth token is good for 20 minutes.  SSH will block new X11 connections after 20 minutes. To avoid this on Linux or OS X, run ssh -Y instead, or set the option ForwardX11Trusted yes in your ~/.ssh/config.

In Windows, use Cygwin/X to provide a Linux-like environment.  Then run ssh -Y or set the option in your ~/.ssh/config file.

Back to Top

Nyx Cloud

The Nyx Cloud hosting system allows launch and configuration of your own Virtual Machine servers. It is a private UM/CCS Cloud system powered by the OpenStack cloud software, which offers the IaaS (Infrastructure-as-a-Service) resource management. It is available to registered CCS Users and resource allocations are Project-based.

The Nyx Cloud is currently in beta using OpenStack cloud software “kilo” version, released April 30, 2015.

Nyx Cloud Environment

Nyx Cloud Virtual machine (VM) instances are grouped into Projects, which reside on dedicated private virtual networks (subnets). Through the dashboard, users can start and customize their own VMs. Nyx VMs can be single- or multiple-CPU servers and can be shut down or restarted as needed. Several bootable images are available, including configurations such as LAMP and MEAN in CentOS 6 and CentOS 7. Snapshots of Projects can be taken for replication and backup. Floating IP addresses are available for SSH connections to Nyx instances from outside the Project network.

Advanced storage features currently undergoing testing include block and object storage. VM instances can be started with dedicated block storage or attached to existing block storage. Object (distributed) storage, which allows for data access via HTTP, is also available. Contact Advanced Computing for assistance with these features.

Account requests

Nyx Cloud users must have an active CCS account, access to secure University of Miami networks, a modern network browser (to access the dashboard), and a terminal program (to use SSH key pairs).

Request a Nyx Cloud account and project from hpc@ccs.miami.edu. Include your CCS username and the name or acronym of the project you would like to create.

Pre-configured images

Pre-configured VM images available on Nyx currently include two basic CentOS and several software application groups. The CentOS default username is clouduser.

  • Centos-6.7-x86_64: CentOS 6.7 basic Linux
  • Centos-7.2-x86_64: CentOS 7.2 basic Linux
  • Centos-6.7-x86_64-MEAN: Mongo/Express/Angular/NodeJS + Solr
    • Mongodb version 3.2.4 – port 27017, bound to localhost only in /etc/mongod.conf
    • Express, Angular – installed through rpm as node modules in /usr/lib/node_modules
    • Nodejs version 5.7.1
    • Solr version 5.5.0 – user solr on port 8983, installed in /opt/solr
    • Java version 1.8.0_74 – installed in /opt/jdk1.8.0_74
  • Centos-6.7-x86_64-JAVA: Java8/Tomcat/PostgreSQL/Solr
    • Java version 1.8.0_74 – installed in /opt/jdk1.8.0_74
    • Tomcat version 7.0.67 – user tomcat on port 8080, installed in /opt/apache-tomcat-7.0.67
    • PostgreSQL version 8.4.20 – port 5432, bound to localhost only
    • Solr version 5.5.0 – user solr on port 8983, installed in /opt/solr
  • Centos-6.7-x86_64-LAMP: Linux/Apache/HTTPD/MySQL/PHP + Solr
    • Apache httpd version 2.2.15 – user apache on port 80
    • MySQL server version 5.1.73 – user mysql on port 8080
    • PHP version 5.3.3
    • Java version 1.8.0_74 – installed in /opt/jdk1.8.0_74
    • Solr version 5.5.0 – user solr on port 8983, installed in /opt/solr

Back to Top

Nyx Cloud Usage

Log into the Nyx Cloud dashboard with your account: http://nyx.ccs.miami.edu/dashboard/auth/login

nyx_login

The dashboard will initially display your Nyx Projects and the workbench (left menu). Project selection and account information are available from the top bar. In the event of a login error following a timeout, clear browser cookies and other plugin data to try again.

nyx_overview

Creating SSH key pairs

Nyx Cloud users authenticate on VM instances with SSH using key pairs. These key pairs must be generated via the dashboard.

  1. From the left menu, navigate to “Project->Compute->Access & Security”
  2. Choose the Key Pairs tab

    nyx_keys

  3. Click on the Create Key Pair button (upper right) to create a new secure key pair (this will open a pop-up window).

    nyx_create_keys

  4. Name your key pair, then click the Create Key Pair button to download your private key (.pem extension).
    You will not be able to download this file again.
  5. Save your private key to a known location.
    If you lose your private key, you will need to generate a new key pair.
  6. Update your private key file permissions so only you can access the file:
    [localmachine: ~]$ chmod 600 mykey.pem

    If using a terminal emulator like PuTTY in a Windows environment: Terminal emulator (PuTTY) SSH keypair instructions

Creating virtual machine instances

Have at least one Nyx Cloud key pair established before creating VMs.

  1. From the left menu, navigate to “Project->Compute->Instances

    nyx_instances2

  2. Click on the Launch Instance button (upper right) to create a new instance (this will open a pop-up window).
  3. On the Details tab, enter or select your VM instance information:

    nyx_launch_instance

    • Instance Name: enter a name for your VM instance
    • Flavor: select a flavor from the drop-down menu to view Flavor Details (right)
    • Instance Count: 1, or more if necessary
    • Instance Boot Source: select Boot from image
    • Image Name: selet a pre-configured image from the drop-down menu
  4. On the Access & Security tab, control access:
    • Key Pair: select a Nyx Cloud key
    • Security Groups: check default (and/or others as needed)
  5. On the Networking tab, ensure Selected networks is defined (and/or add as needed)
  6. Click the Launch button to start your instance. View progress on the Instances page.

Controlling virtual machine instances

From the dashboard workbench (left menu), navigate to “Project->Compute->Instances“. Under the Actions column, select an option from the drop-down menu for that instance. Before shutting off an instance, we suggest first disassociating any floating IPs.

nyx_instances

  • Soft Reboot Instance: reboot instance
  • Shut Off Instance: power off instance temporarily, preserving current state
  • Start Instance: start or re-start an instance
  • Disassociate Floating IP: release floating IP address (static IPs remain for the duration of the instance)
  • Terminate Instance: delete instance completely
    Terminated instances are not recoverable. Termination removes the virtual machine, all associated block volumes (storage), and disassociates static IP addresses.

Enabling outside SSH access

To enable SSH connections to VM instances from outside the Nyx Project network, associate floating IP addresses. Connect using the floating point IP address directly.

  • Under the Actions column on the Instances page, choose Associate Floating IP from the drop-down menu (this will open a pop-up window).

    nyx_floating_IPs

  • IP Address: select an available IP address from the drop-down menu
  • Port to be associated: the default will be the VM instance used to open the window
  • Click the Associate button (bottom right)
  • On the Instances page, note the assigned address in the IP column.

Accessing instances via SSH

To access a new VM instance from outside, use the Nyx default clouduser and the key pair assigned to the instance. Subsequent external SSH connections can use usernames and passwords, if established.

  1. SSH as clouduser with your corresponding private key can access it:
    [localmachine: ~]$ ssh -i mykey.pem clouduser@floating.point.IP.address

    If using a terminal emulator like PuTTY in a Windows environment: Terminal emulator (PuTTY) SSH keypair instructions

  2. Change your password:
    [clouduser@nyxtest ~]$ sudo passwd clouduser
    Changing password for clouduser.
    New password:  new_password
    Retype new password:  new_password
    passwd:  all authentication tokens update successfully.
  3. Check your password:
    [clouduser@nyxtest ~]$ ssh clouduser@floating.point.IP.address
    clouduser@floating.point.IP.address's password:  password
    [clouduser@nyxtest ~]$ 

Controlling IP ports

Certain software may need ports explicitly opened and specific IP protocols established. Navigate to “Project->Compute->Access & Security” and choose the Security Groups tab. Under the Actions column, choose Manage Rules from the drop-down menu for the appropriate security group. Create your own I/O rules by clicking the Add Rule button on the upper right.

nyx_security_groups

Back to Top

Suggest Edit

Advanced Computing Documentation & User Guides contains detailed information about our systems. User guides for the Pegasus Supercomputer — including high-performance clusters, software, and job submissions — and the Nyx Cloud are grouped by category. Access specific topics by clicking the arrows next to menu items and search all documentation pages with the search bar.

For help using Linux, see our Pegasus Linux User Guides.